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Preface 


This document contains the proceedings of the Training Workshop on 
Nondeterministic Approaches and their Potential for Future Aerospace Systems held at 
NASA Langley Research Center, Hampton, Virginia, May 30-31, 2001. The workshop was 
jointly sponsored by Old Dominion University and NASA. Workshop attendees came from 
NASA, other government agencies, industry, and universities. The objectives of the 
workshop were to review the diverse activities in nondeterministic approaches, uncertainty 
management methodologies, reliability assessment and risk management techniques, and to 
identify their potential for future aerospace systems. 


Ahmed K. Noor 
Old Dominion University 
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INTRODUCTION 


Increasingly more complex systems are being built and conceived by high-tech 
industries. Engineers are asked to design faster, and to insert new technologies into these 
systems. Increasing reliance is being made on modeling, simulation and virtual prototyping 
to find globally optimal designs that take uncertainties and risk into consideration. 
Conventional computational and design methods are inadequate to handle these tasks. 
Therefore, intense effort has been devoted in recent years to nontraditional methods for 
solving complex problems with system uncertainties. 

An attempt is made in this overview to give broad definitions to the terms and to set 
the stage for the succeeding presentations. The presentation is divided into four parts (see 
Figure 1). In the first part, examples of future aerospace systems are given, along with some 
of their major characteristics and design drivers. The second part describes the synergistic 
coupling of the key technologies that can significantly enhance the modeling and simulation 
technologies. The third part describes the future research and learning environments required 
for the realization of the full potential of nondetermini stic approaches. The fourth part lists 
the objectives of the workshop and some of the sources of information on nondeterministic 
approaches. 


• Characteristics of Future Aerospace Systems 



r ;X<'. ■: ■o.y-s'V s ? : ■ 

• Modeling and Simulation Technologies 


• Research and Learning Environments 

i|r~ 



• Workshop 

Si 


Figure 1 


3 


EXAMPLES OF FUTURE AEROSPACE SYSTEMS AND SOME OF THEIR 

CHARACTERISTICS 


The realization of NASA’s ambitious goals in aeronautics and space with the current 
national budget constraints will require new kinds of aerospace systems and missions that use 
novel technologies and manage risk in new ways. Future aerospace systems must be 
autonomous, evolvable, resilient, and highly distributed. Two examples are given in Figure 
2. The first is a biologically inspired aircraft with self-healing wings that flex and react like 
living organisms. It is built of a multifunctional material with fully integrated sensing and 
actuation, and unprecedented levels of aerodynamic efficiencies and aircraft control. The 
second is an integrated human-robotic outpost, with biologically inspired robots. The robots 
could enhance the astronaut’s capabilities to do large-scale mapping, detailed exploration of 
regions of interest, and automated sampling of rocks and soil. They could enhance the safety 
of the astronauts by alerting them to mistakes before they are made, and letting them know 
when they are showing signs of fatigue, even if they are not aware of it. 
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DEFINITIONS OF UNCERTAINTY AND A BRIEF HISTORICAL ACCOUNT OF 

UNCERTAINTY MODELING 


Uncertainty is an acknowledged phenomenon in the natural and technological worlds. 
Engineers are continually faced with uncertainties in their designs. However, there is no 
unique definition of uncertainty. A useful functional definition of uncertainty is: the 

information/knowledge gap between what is known and what needs to be known for optimal 
decisions, with minimal risk. 

Prior to the twentieth century, uncertainty and other types of imprecisions were 
considered to be unscientific, and therefore, not addressed. It was not until the beginning of 
the twentieth century that statistical mechanics emerged and was accepted as a legitimate 
area of science. It was taken for granted that uncertainty is adequately captured by 
probability theory. It took sixty years to recognize that the conceptual uncertainty is too deep 
to be captured by probability theory alone and to initiate studies of non-probabilistic 
manifestations of uncertainty, as well as their applications in engineering and science. In the 
last two decades, significant advances have been made in uncertainty modeling, the level of 
sophistication has increased, and a number of software systems have been developed. 
Among the recent developments are the perception-based information processing and 
methodology of computing with words (Figure 3). 


TYPES OF UNCERTAINTIES 


A number of different uncertainty representations and classifications have been 
proposed. Among these classifications are the three-type classification - statistical, model, 
and fundamental uncertainties; the two type classification - uncertainty of information and 
uncertainty of the reasoning process; and the six-type classification (see Figure 4): 

■ Probabilistic uncertainty, which arises due to chance or randomness, 

■ Fuzzy uncertainty due to linguistic imprecision (e.g., set boundaries are not 
sharply defined), 

■ Model uncertainty which is attributed to lack of information about the model 
characteristics, 

■ Uncertainty due to limited ( fragmentary ) information available about the system 
(for example, in the early stage of the design process), 

■ Resolutional uncertainty which is attributed to limitation of resolution (e.g., 
sensor resolution), and 

■ Ambiguity (i.e., one to many relations). 



Figure 4 





MANAGING UNCERTAINTIES 


While completely eliminating uncertainty in engineering design is not possible, 
reducing and mitigating its effects have been the objectives of the emerging field of 
uncertainty management. The field draws from several disciplines including statistics, 
management science, organization theory, and inferential thinking (see Figure 5). 
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NONDETERMINISTIC ANALYSIS APPROACHES 


Depending on the type of uncertainty and the amount of information available about 
the system characteristics and the operational environments, three categories of 
nondeterministic approaches can be identified for handling the uncertainties. The three 
approaches are (see Figure 6): probabilistic analysis, fuzzy-set approach, and set theoretical, 
convex (or anti-optimization) approach. In probabilistic analysis, the system characteristics 
and/or the source variables are assumed to be random variables (or functions), and the joint 
probability density functions of these variables are selected. The main objective of the 
analysis is the determination of the reliability of the system. 

If the uncertainty is because of a vaguely defined system and/or operational 
characteristics, imprecision of data, and subjectivity of opinion or judgment, fuzzy-set 
treatment is appropriate. Randomness describes the uncertainty in the occurrence of an event 
(such as damage or failure). 

When the information about the system and/or operational characteristics is 
fragmentary (e.g., only a bound on a maximum possible response function is known), then 
convex modeling is practical. Convex modeling produces the maximum or least-favorable 
response and the minimum or most favorable response of the system under the constraints 
within the set-theoretic description. 
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ENHANCING THE MODELING AND SIMULATION TECHNOLOGIES 


The synergistic coupling of nondeterministic approaches with a number of key 
technologies can significantly enhance the modeling and simulation capabilities and meet the 
needs of future complex systems. The key technologies include: Virtual product 

development for simulating the entire lifecycle of the engineering system, reliability and risk 
management, intelligent software agents, knowledge and information, high performance 
computing, high capacity communications, human computer interfaces, and human 
performance. 



Figure 7 
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VIRTUAL PRODUCT DEVELOPMENT 


Current virtual product development (VPD) systems have embedded simulation 
capabilities for the entire lifecycle of the product. As an example, the top-level system 
process flow for a space transportation system is shown in Figure 8. In each phase 
uncertainties are identified and appropriate measures are taken to mitigate their effects. 
Information Technology will change the product development from a sequence of distinct 
phases into a continuous process covering the entire lifecycle of the product with full 
interplay of information from beginning to end and everywhere throughout. 



Figure 8 
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RELIABILITY ASSESSMENT 


Reliability is defined as the probability that a component (or a system) will perform 
its intended function without failure for a specified period of time under designated operating 
conditions. Failure rate or hazard rate is an important function in reliability analysis since it 
provides a measure of the changes in the probability of failure over the lifetime of a 
component. In practice, it often exhibits a bathtub shape (see Figure 9). 

Reliability assessment includes: selection of a reliability model, analysis of the 

model, calculation of the reliability performance indices, and evaluation of results, which 
includes establishment of confidence limits and decision on possible improvements. 
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RISK MANAGEMENT PROCESS 


Risk is defined as the uncertainty associated with a given design, coupled with its 
impact on performance, cost and schedule. Risk management is defined as the systems 
engineering and program management tools that can provide a means to identify and resolve 
potential problems. 

The risk management process includes the following task (Figure 10): 

■ Risk planning', development of a strategy for identifying risk drivers. 

■ Risk identification: identifying risk associated with each technical process. 

■ Risk analysis : isolating the cause of each identified risk category and 

determining the effects. 

The combination of risk identification and risk analysis is referred to as risk 
assessment. 

■ Risk handling : selecting and implementing options to set risk at acceptable 
levels. 

■ Risk monitoring: systematically tracking and evaluating the performance of 
risk handling actions. 



Figure 10 


12 






ROBUSTNESS 



Robust Design 


Non-Robust 

Design 


Performance 


Var. in 

Components 


Var. in 

Components 


Var. in 

Environment 


Var. in 

Environment 


Robustness is defined as the degree of tolerance to variations (in either the 
components of a system or its environment). A robust ultra-fault-tolerant design of an 
engineering system is depicted in Figure 11. The performance of the system is relatively 
insensitive to variations in both the components and the environment. By contrast, a non- 
robust design is sensitive to variations in either or both. 


Figure 11 
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TERAMAC CONFIGURABLE CUSTOM COMPUTER 


An example of a robust, ultra-fault-tolerant system is the Teramac Computer which is 
a one Tera Hertz massively parallel experimental computer built at Hewlett-Packard 
Laboratories to investigate a wide range of computational architectures (Figure 12). It 
contains 22,000 (3%) hardware defects, any one of which could prove fatal to a more 
conventional machine. It incorporates a high communication bandwidth that enables it to 
easily route around defects. It operates 100 times faster than a high-end single processor 
workstation (for some of its configurations). 
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KEY COMPONENTS OF ADVANCED SIMULATION AND MODELING 

ENVIRONMENTS 


The realization of the full potential of nondetermini stic approaches in modeling and 
simulation requires an environment that links diverse teams of scientists, engineers, and 
technologists. The essential components of the environment can be grouped into three 
categories (Figure 13): intelligent tools and facilities, nontraditional methods, and advanced 
interfaces. 



Figure 13 
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INTELLIGENT TOOLS AND FACILITIES 


These include high fidelity - rapid modeling, lifecycle simulation and visualization 
tools, synthetic immersive environment; automatic and semiautomatic selection of software 
and hardware platforms; computer simulation of physical experiments and remote control of 
these experiments. In all of these tools, extensive use should be made of intelligent software 
agents and information technology (Figure 14). 
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ADVANCED HUMAN/COMPUTER INTERFACES 


Although the WIMP (windows, icons, menus, pointing devices) paradigm has 
provided a stable global interface, it will not scale to match the myriad from factors and uses 
of platforms in the future collaborative distributed environment. Perceptual user interfaces 
(PUT) are likely to meet those needs. PUI’s integrate perceptive, multimodal and multimedia 
interfaces to bring human capabilities to bear on creating more natural intuitive interfaces. 
They enable multiple styles of interactions, such as speech only, speech and gesture, vision, 
and synthetic sound, each of which may be appropriate in different applications (Figure 15). 
These new technologies will enable broad uses of computers as assistants or agents that will 
interact in more human-like ways. 



• Integrates perceptive, multimodal and multimedia 
interfaces to bring human capabilities to bear on 
creating more natural and intuitive interfaces 


• Enables multiple styles of interactions and broad 
uses of computers as assistants 


Figure 15 


17 






NONTRADITIONAL METHODS 


These include multi-scale methods, strategies for highly coupled multi-physical 
problems, and nondeterministic approaches for handling uncertainty in geometry, material 
properties, boundary conditions, loading and operational environments (Figure 16). 



Figure 16 
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PRINCIPLE OF COMPLEXITY 


One of the important consequences of uncertainty is its effect on precision. Three 
types of models can be identified depending on the complexity and the precision, namely: 
mathematical models, model-free methods, and fuzzy systems (Figure 17). In a typical 
complex system a combination of the three should be used. As the uncertainty and/or 
complexity of an engineering system increases, the ability to predict its response diminishes, 
until a threshold is reached beyond which precision and relevance become almost mutually 
exclusive. Consider, for example, numerical simulations in which sophisticated 
computational models are used for predicting the response, performance, and reliability of 
the engineering system, but the system parameters are little more than guesses. Such 
simulations can be characterized as Correct but Irrelevant Computations (CBIC); that is, 
forcing precision where it is not possible. 
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BOUNDING UNCERTAINTIES IN SIMULATION MODELS 


Current synthesis approaches of simulation models involve a sequence of four phases 
(Figure 18). 

First - selection of the models, which includes decisions about modeling approach, 
level of abstraction, and computational requirements. The complexities arise due to: 

■ Multiconstituents, multiscale, and multiphysics material modeling, 

* Integration of heterogeneous models, 

Second - parameter identification. Data reduction techniques are used which 
incorporate uncertainties, 

Third - model updating, or reducing uncertainty by improving either the model 
characteristics or the model itself, and 

Fourth - Validation, in the sense of confirming that the model is an accurate 
representation of the real system. 
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QUALITY CONTROL AND UNCERTAINTY MANAGEMENT IN THE 
MODELING AND SIMULATION OF COMPLEX SYSTEMS 


The estimation of total uncertainty in the modeling and simulation of complex 
systems involves: a) identification and characterization of the sources of uncertainty, 

variability and error; b) uncertainty propagation and aggregation; and c) uncertainty 
quantification (Figure 19). 

Herein uncertainty is defined as a deficiency in any phase of the modeling process 
due to lack of knowledge (model form or reducible uncertainty) increasing the knowledge 
base can reduce the uncertainty. The term variability is used to describe inherent variation 
associated with the system or its environment (irreducible or stochastic uncertainty) 
variability is quantified by a probability or frequency distribution. An error is defined as a 
recognizable deficiency that is not due to lack of knowledge. An error can be either 
acknowledged (e.g., discretization or round-off error), or unacknowledged (e.g., 
programming error). 



Figure 19 
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VERIFICATION AND VALIDATION OF NUMERICAL SIMULATIONS 


Quantifying the level of confidence, or reliability and accuracy of numerical 
simulations has recently received increased levels of attention in research and engineering 
applications. During the past few years, new technology development concepts and 
terminology have arisen. Terminology such as virtual prototyping and virtual testing is now 
being used to describe computer simulation for design, evaluation and testing of new 
engineering systems. 

The two major phases of modeling and simulation of an engineering system are 
depicted in Figure 20. The first phase involves developing a conceptual and mathematical 
model of the system. The second phase involves discretization of the mathematical model, 
computer implementation, numerical solution and representation or visualization of the 
solution. In each of these phases there are uncertainties, variabilities and errors. 

Verification and validation are the primary methods for building and quantifying 
confidence in numerical simulations. Verification is the process of determining that a model 
implementation accurately represents the conceptual/mathematical model and the solution to 
the model. Correct answer is provided by highly accurate solutions. Validation is the 
process of determining the degree to which a model is an accurate representation of the real 
system. Correct answer is provided by experimental data. 


Figure 20 


DESIGN OF EXPERIMENTS 


These are systematic techniques for investigating (all possible) variations in system 
performance due to changes in system variables. 

Two categories of system variables can be identified, namely, a) Inner-array 
variables, which are controllable; and, b) outer-array variables (also called noise 
factors), which are functions of environmental conditions, and are uncontrollable. 

Three categories of techniques can be identified: Regression analysis, statistical 
methods and Taguchi’s method (Figure 21). 

In Taguchi’s method, the controllable variables are selected in such a way as to 
dampen the effect of the noise variables on the system performance. The method was 
originally developed as an industrial total quality control approach. Subsequently, it has 
found several other applications, including design optimization through variability reduction. 
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EDUCATION, TRAINING AND LEARNING 


There has long been a philosophical gap between education and training. The goal of 
education was to impart high-level cognitive skills that would underpin lifelong learning. 
The goal of training was to bring performance up to a level that would let people successfully 
achieve tasks. Recently, however, began to emphasize the skills involved in lifelong 
learning, as evidence by continual-growth workshops and online training facilities on the 
Internet. In a sense, both education and training objectives fit in the larger classification of 
learning objectives (see figure 22). 
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LEARNING OBJECTIVES, INSTRUCTIONAL MODELS AND TECHNOLOGIES 


The desired outcome of learning can range from information transfer to skill and 
knowledge acquisition to the more ambitious goal of development of critical thinking and 
creativity skills. The instructional model and method used for accomplishing these goals 
vary from instructor-centered, learner-center to learning-team centered. In the learner- 
centered model, the learner is at the center of the learning process, and calls on many 
information sources. Learning-team center models include virtual classrooms and web-based 
distance learning models. The technologies employed in the three models are distribution, 
interactive and collaborative technologies, respectively (See Figure 23). 
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LEARNING NETWORKS 


The convergence of computing, communication and information technologies is 
providing opportunities for creating effective environments for life-long learning through 
expanding the concept of a university which is, typically limited to a campus, to that of a 
learning network (Figure 24). In such a network, the classrooms are augmented by e- 
leaming facilities (e.g., virtual classrooms); the libraries are expanded into intelligent 
knowledge repositories (with digital libraries and intelligent search and information 
visualization capabilities); the physical test and experimental facilities are augmented with 
access to more elaborate facilities at government labs, along with computer simulation of 
these facilities; and Immersive telepresence technology is used to provide interaction with 
geographically dispersed instructors and learners at other locations. 



Figure 24 
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ACTIVITIES OF LEARNING NETWORKS 


The learning networks can significantly enhance the effectiveness of engineering 
education, by changing the way three of the major functions of a university are carried out, 
namely, development of content for courses, packaging courses into curricula and programs, 
and delivery of these programs to learners (Figure 25). 

Each course is divided into self-contained learning modules, and a consortium is 
established for generating the best content for each of the learning modules. Advanced 
instructional technology; modeling, simulation and visualization facilities and authoring tools 
are used in the development of modules. 

The learning modules are then packaged into disciplinary and interdisciplinary 
courses and training programs to satisfy the needs of diverse groups. 

The packaged modules are presented to individuals as well as groups of learners. 
Collaboration and interaction is made available at many levels, both synchronous and 
asynchronous. 
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ADVANCED LEARNING ENVIRONMENTS 


In order to meet the life long learning demands of the future and broaden the 
awareness among the researchers and engineers of nondeterministic approaches, three 
categories of learning environments are needed; namely, expert led group learning 
environment; self paced individual learning environment; and collaborative learning 
environment (Figure 26). The three environments, in combination, can reduce the time and 
cost of learning, as well as sustain and increase worker competencies in high tech 
organizations. 

The human instructors in these environments will serve many roles, including 
inspiring, motivating, observing, evaluating, and steering the learners, both individually and 
in distributed teams. 


Figure 26 







EXPERT LED LEARNING ENVIRONMENT 


The human instructors in expert-led distributed learning in a virtual environment 
serve as coaches, guides, facilitators, and course managers. Their presentations focus on a 
broad overview of the topic and its diverse applications (Figure 27), and end with more 
penetrating, what-if questions that can enhance the critical thinking and creativity of the 
learners. Elaborate visualization and multimedia facilities are used in the presentations. 
Routine instructional and training tasks are relegated to the self-paced individual learning 
environment. 
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Figure 27 
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SELF-PACED LEARNING ENVIRONMENT 


The individual learning environment engages the learner and provides a high degree 
of tailored interactivity. It can be used for self-paced instruction of routine material not 
covered in the lecture. Using virtual instructors assigned by the human instructors can 
enhance such instruction. It can be used to study the effect of various types of uncertainties 
on the system performance using advanced visualization, multimedia and multisensory 
immersive facilities. The individual learning environment can serve to carry out numerical 
and virtual experiments - computer simulation of physical experiments (Figure 28). 
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COLLABORATIVE / DISTRIBUTED LEARNING ENVIRONMENT 


Collaborative learning environments teach teamwork and group problem solving. 
Instructors and learners can be geographically dispersed. Eventually, they can be brought 
together through immersive telepresence facilities to share their experiences in highly 
heterogeneous environments involving different computing platforms, software and other 
facilities, and they will be able to work together to design complex engineering systems 
beyond what is traditionally done in academic settings. Because participants can be virtually 
collocated without leaving their industry and government laboratories, collaborative learning 
environments can enable the formation of learning networks linking universities, industry 
and government labs. The ultimate goal of these learning facilities is to create an intellectual 
environment where academic and experiential learning are effectively and efficiently co- 
mingled. In such an environment, academic rigor is learned in concert with professional job 
performance, and academic complexities are addressed within the industrial concern (Figure 
29). 



Figure 29 


31 






VIRTUAL CLASSROOM 


Online training and virtual classrooms are typically used to provide learning 
environments with custom self-instruction, flexible tutorial support, and choice of both the 
place and time of learning. Three categories of facilities are used in these environments; 
namely: instruction , including multimedia lectures, links to other resources and tools for 
searching, browsing, and using archived knowledge; communication , including email, 
UseNet, chat centers, video and Internet conferencing; and course management and 
performance evaluation (Figure 30). 
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The realization of the full potential of nondetermini Stic approaches in the design and 
development of future complex systems requires, among other things, the establishment of 
research and learning networks. The networks connect diverse, geographically dispersed 
teams from NASA, other government labs, university consortia, industry, technology 
providers, and professional societies (Figure 31). 


Figure 31 
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EVOLUTION OF NEW TECHNOLOGY 


The nondeterministic approaches and their associated technologies, as any other 
technology, have gone through three phases. The first is that of naive euphoria - unrealistic 
expectations resulting from overreaction to immature technology. The second is cynicism, or 
frustration associated with unmet expectations. The third is that of realistic expectations - 
gradually realizing the true benefits from the technologies (Figure 32). 



Figure 32 
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OBJECTIVES AND FORMAT OF WORKSHOP 


The objectives of the workshop are to: a) provide a broad overview of 

nondeterministic approaches, uncertainty management methodologies, reliability assessment 
and risk management techniques, and b) identify the potential of these technologies to future 
aerospace systems (Figure 33). The workshop, including sixteen presentations and three 
exhibits, illuminate some of the key issues in nondeterministic approaches and provide fresh 
ideas for future research and development (Figure 34). 

Objectives and Format of Workshop 

Objectives 

• Overview of diverse activities in 
Nondeterministic Approaches 

• Identify potential for future 
aerospace systems 

Format 

• 16 presentations, 7 sessions 

• 3 Exhibits 



Figure 33 
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INTRODUCTION AND OBJECTIVE 


Future systems will require engineers to design them, test their performance, and 
assess their robustness and vulnerability in a simulated environment. Simulation requires 
validated building blocks for materials behavior, physical laws, environment-system 
interaction, unit performance, and system performance. At every level and stage of the 
simulation process, verification and validation are needed that should include ignorance 
analysis, and uncertainty analysis and modeling. Example systems include our nuclear 
weapon stockpile, space stations, satellites, space missions, etc. 


Introduction 


> Engineering Systems 

- Complex engineering systems and reliance on 
simulation, such as: 

• Advanced systems (e.g., Mobile offshore base); 

• Power plants; and aerospace & space mission 
systems 

require modeling and assessment of knowledge 
and ignorance. 


> Objective 

- Adapt and develop quantitative models and 
measures suitable for prediction and decision- 
based design of complex engineering systems 
under conditions of uncertainty or ignorance. 



Figure 1 


39 



SYSTEM BREAKDOWN OF SHIPS 


An example breakdown is provided for the functions of a ship. A work breakdown 
structure is also provided for illustration purposes. 
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SYSTEM BREAKDOWN OF DAMS 


An example breakdown is provided for the functions of a dam. A work breakdown 
structure is also provided for illustration purposes. 
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LIFECYCLE OF NASA ENGINEERING SYSTEMS 


The phases of the lifecycle of NASA systems are provided in this viewgraph covering 
pre-phase and phase A. The details of these phases are listed. 


Lifecycle of NASA Engineering 
Systems 


> Pre-phase A. Advanced Studies 

> identify missions consistent with the NASA charter 

> identify and involve users 

> perform preliminary evaluations of possible missions 

> Phase A. Conceptual Design Studies 

> preparation of mission needs statements 

> development of preliminary system requirements 

> identification of alternative operations and logistics concepts 

> identification of project constraints and system boundaries 

> consideration of alternative design concepts 

> demonstrating that credible, feasible designs exist 
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LIFECYCLE OF NASA ENGINEERING SYSTEMS 


The phases of the lifecycle of NASA systems are provided in this viewgraph covering 
phases B, C, D, E, and F. The details of these phases are listed. 


Lifecycle of NASA Engineering 
Systems 

> Phase B. Concept Definition (selected items) 

> reaffirmation of the mission needs statement 

> preparation of a program initiation agreement 

> preparation of a system engineering management plan 

> preparation of a risk management plan 

> Phase C. Design and Development 

> Phase D. Fabrication, Integration, Test and 
Certification 

> Phase E. Pre-Operations 

> Phase F. Operations and Disposal 



Figure 5 
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MODELING & SIMULATION OF ENGINEERING SYSTEMS 


Modeling and simulation of engineering systems consists of several steps. These 
steps are key to the processes of model qualification, verification and validation. The 
processes of qualification, verification and validation are shown in the figure. The 
verification process consists of three stages: conceptual model verification, design 
verification, and code verification. The verification can be done by comparison and test of 
agreement between the computational model and solution, and results from benchmark 
(analytical or very accurate numerical solutions) of simplified model problems. The 
validation consists of two stages: conceptual model validation, and results validation that can 
be done by expert opinion solicitation. The objective herein is to adapt and develop 
quantitative models and measures suitable for prediction and decision-based design of 
complex engineering systems under conditions of uncertainty or ignorance. 


Modeling & Simulation of 
Engineering Systems 


> Conceptual modeling of a real system 

> Mathematical modeling of the conceptual designs 

> Discretization and algorithm selection 

> Computer programming 

> Numerical solution 

> Representation of the 
numerical solution 
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KNOWLEDGE NEEDS FOR SOLVING PROBLEMS AT THE SYSTEM LEVEL 


Problem definition and abstraction are key elements to problem solving and 
knowledge construction. 


Knowledge Needs for Solving 
Problems at the System Level 

> “The mere formulation of a problem 
is often far more essential than its 
solution..." Albert Einstein 

> “What we observe is not nature 
itself, but nature exposed to our 
method of questioning." 

Werner Karl Heisenberg 
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KNOWLEDGE CATEGORIES 


A breakdown of knowledge categories, objects of knowledge and knowledge sources 
are provided herein. The knowledge base about a system is a mixture of truth and fallacy. 
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KNOWLEDGE, INFORMATION, OPINIONS, AND 
EVOLUTIONARY EPISTEMOLOGY 


The definition of knowledge, information, opinions, and evolutionary epistemology 
are provided. 


Knowledge, Information, Opinions, 
and Evolutionary Epistemology 






Figure 9 





KNOWLEDGE DEFINITION AND CHARACTERISTICS 


Knowledge is defined and its characteristics are provided. Knowledge is relative with 
potential biasedness and time asymmetry. 


Knowledge 

Definition and Characteristics 
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CLASSIFICATION OF IGNORANCE 


Ignorance can be classified into three groups with a subsequent level of blind 
ignorance and conscious ignorance. The state of ignorance for a person or society can be 
unintentional or deliberate due to an erroneous cognition state and not knowing relevant 
information, or ignoring information and deliberate inattention to something for various 
reasons such as limited resources or cultural opposition, respectively. The latter type is a 
state of conscious ignorance which is not intentional, and once recognized evolutionary 
species try to correct for that state for survival reasons with varying levels of success. The 
former ignorance type belongs to the blind ignorance category. Therefore, ignoring means 
that someone can either unconsciously or deliberately refuse to acknowledge or regard, or 
leave out an account or consideration for relevant information (di Carlo 1998). These two 
states should be treated in developing a hierarchal breakdown of ignorance. 


Classification of Ignorance 
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Figure 1 1 
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CLASSIFICATION OF IGNORANCE 


Ignorance can be viewed to have a hierarchal classification based on its sources and 
nature as shown in the figure. Ignorance can be classified into two types, blind ignorance 
(also called meta-ignorance), and conscious ignorance (also called reflective ignorance). 
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BLIND IGNORANCE 


Blind ignorance includes not knowing relevant know-how, objects-related 
information, and relevant propositions that can be justified. The unknowable knowledge can 
be defined as knowledge that cannot be attained by humans based on current evolutionary 
progressions, or cannot be attained at all due to human limitations, or can only be attained 
through quantum leaps by humans. Blind ignorance also includes irrelevant knowledge that 
can be of two types: (1) relevant knowledge that is dismissed as irrelevant or ignored, and (2) 
irrelevant knowledge that is believed to be relevant through non-reliable or weak justification 
or as a result of ignoratio elenchi. The irrelevance type can be due to untopicality, taboo, and 
undecidability. Untopicality can be attributed to intuitions of experts that could not be 
negotiated with others in terms of cognitive relevance. Taboo is due to socially reinforced 
irrelevance. Issues that people must not know, deal with, inquire about, or investigate define 
the domain of taboo. The undecidability type deals with issues that cannot be designated true 
or false because they are considered insoluble, or solutions that are not verifiable, or as a 
result of ignoratio elenchi. A third component of blind ignorance is fallacy that can be 
defined as erroneous beliefs due to misleading notions. 


Blind 

Ignorance 




Ignorance 

i 


Conscious Ignorance 


| Blind Ignorance ] 


Inconsistency] | Incompleteness | [Fallacy! lUnknownabie] | Irrelevance] 


Confusion 


f Inaccuracy | 


| Unknowns] 


| Untopicality] 


| Undecidabiiity 


[Conflict! 


| Uncertainty! 1 Absence 1 


Taboo 1 






|Vaguenessj 1 Coarseness") 


[simplifications! 
_£Z 


Nonspecificity 1 { Unspecificity | 


ZZL. 


Blind Ignorance: Ignorance of i-a s 
self-ignorance or called meta-ignorance. 

Fallacy: erroneous belief due to misleading notions 


> Unknowable: Knowledge that cannot be attained by 
humans based on current evolutionary progressions or 
limitations, or can only be attained through quantum 
leaps by humans . 


> Irrelevance: Ignoring something. 

> Untopicality: attributed to intuitions of experts that are 
negotiated with others in terms of cognitive relevance. 

> Taboo: due to socially reinforced irrelevance. 

> Undecidability: deals with issues that are considered 
insoluble or solutions that are not verifiable. 

CTSh'i 



Figure 13 


51 



CONSCIOUS IGNORANCE 


It has two primary components of inconsistency and incompleteness as detailed in the 

figure. 
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Kurt Godel (1906-1978) showed that a 
logical agent could not be both consistent 
and complete; and could not prove itself 
complete without proving itself 
inconsistent and vise versa. 





| Unspecificity] 


Conscious Ignorance: A recognized self- 
ignorance through reflection. 

> Inconsistency 


- Confusion (Wrongful substitutions) 

- Conflict (Contradictory assignments or substitutions) 

- Inaccuracy (Bias and distortion in degree) 


> Incompleteness 

- Unknowns (The difference between the becoming 
knowledge state and current knowledge state) 

- Absence (Incompleteness in kind) 

- Uncertainty (inherent deficiencies with acquired 
knowledge) 

• Ambiguity, Likelihood, Approximations 
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THEORIES TO MODEL AND ANALYZE IGNORANCE TYPES 


This table maps available theories to various ignorance categories. Ayyub (2001) 
provides details on this classification and theories recommended for various categories. 


Theories to Model and Analyze 


Ignorance Types 



Ignorance Type 


Theory 


Confusion & 
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Inaccuracy Ambiguity 


Randomness & 
Sampling 


Simplification 
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Possibility 


Monotone measure 


Interval probabilities 


Interval analysis 


Figure 15 


53 





SYSTEM DEFINITION LIMITATIONS 


Two limitations are provided that correspond to organized and non-structured 
complexity. Ayyub (2001) discusses and demonstrates these limitations. 


Limitations 

(Bremermann 1962): 

“No data processing 
systems, whether artificial 
or living, can process more 
than 2X10 47 bits per second 
per gram of its mass ” 


Bremermann’ s limit 
JO 93 bits 

Pattern Recognition 

k n < 10 93 

For qxq spatial array defining 
n = q 2 cells with k colors 

Human retina of about one 
million cells: 2 1 ’ 000 ’ 000 = 10 300 


mm 
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SELECTED METHODS 


These selected methods were described at the workshop with examples. 


Selected Methods 

> Probability 

S Classical theory 
S Interval probabilities 
s Imprecise probabilities 

> Bayesian methods 

> Fuzzy sets, fuzzy arithmetic, constrained 
fuzzy arithmetic, fuzzy probabilities 

> Rough sets 

> Possibility theory 

> Fuzzy measure theory 

> Dempster-Shafer theory of evidence 

Cl Hi 


Figure 17 
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MONOTONE MEASURES 


Monotone measures provides a generalization of many methods listed in the previous 
viewgraph. 


Monotone Measures 


A monotone measure for a non-empty family A of 
subsets for a given universal set X, is a mapping as 
follows: 

/: A -a [ 0 ,°°] 


For any pair A ; and A 2 e A such that A ; n A 2 = 0, 
super-additive (< cooperative action or synergy between A 1 
and A 2 ): 

f(A J u A 2 ) >f(Aj) +/(A 2 ) 

Additive (no interaction) 

jf(A 7 uA 2 ) -f{Aj) +f(A 2 ) 

Sub-additive ( inhibitory effect or incompatibility ) 



f(Aj u A 2 ) <f(Aj) +/(A 2 ) 



Figure 18 


56 





CLASSIFYING MONOTONE MEASURES 


The relationships between monotone measures and selected theories are listed herein. 
Ayyub (2001) provides details on these relationships. 


Classifying Monotone 
Measures 


• Classical probability theory (crisp sets and 
additive measures). 

• Probability theory based on fuzzy events (fuzzy 
sets and additive measures) 

• Dempster-Shafer theory of evidence and its 
monotone measures of belief and plausibility 
(crisp sets and nonadditive measures). 

• Fuzzified Dempster-Shafer theory of evidence 
(fuzzy sets and nonadditive measures). 
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CLASSIFYING MONOTONE MEASURES 

The relationships between monotone measures and selected theories are listed herein. 
Ayyub (2001) provides details on these relationships. 


Classifying 
Monotone Measures 

• Possibility theory and its monotone measures of 
necessity and possibility (crisp sets and nonadditive 
measures). This possibility theory case is a special 
case of the above Dempster-Shafer theory of 
evidence by requiring underlying events to be 
nested, i.e., A 1 c A 2 c . . . c X. 

• Possibility theory based on fuzzy events (fuzzy sets 
and nonadditive measures). 

• Other cases. A large number of cases can be 
developed based on the nonadditive measures, such 

as imprecise probabilities, and based on rough sets^— 

m Q| : 

Figure 20 
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UNCERTAINTY-BASED INFORMATION: UNCERTAINTY MEASURES 


Uncertainty measures can be used to solve engineering problems, such as regression 
based on minimizing uncertainty, combining expert opinions based on maximizing 
uncertainty, and information consolidation based on the uncertainty invariance principle. 



Uncertainty Theory 

Uncertainty Measure 

Equation 

Uncertainty Type 

Year 


Classical set theory 

U(A) = logj |A| 

(9.1) 

Nonspecificity 

1928 

Fuzzy set theory 

a fh<A) 

uw= ml ^ rA]da 

(9.22) 
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V(r) = ]jP r ' logj ——r 

im 2 11 

(9.26) 

Nonspecificity 

u 

Uncertainty- 
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Evidence theory 

, i»W) iog 2 |A| 

(9.36) 

> 

Nonspecificity 


Probability theory 

H(m) = - £ /»({*}) log 2 

xeX . 

(9.37) 

Strife 

1948 

Evidence theory 

S(m ) = - £ m(A) log 2 £ m W 

AeJ B& 1 A « 

(9-54) 

Strife 

1992 

Possibility theory 

n 1 * 

S(r) = £(r, - ri + i)log 2 - 

i-i 

(9.58) 

Strife 

1992 

Evidence theory 
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Bel 

(9.59) 

Total: N(m) + S(m) 

1992 

Possibility theory 

NS(r) m — ty+OIogj —— 

i»2 

Ln 

i-i 

(9.60) 

Total: N(r) + S(r ) 
r ‘ 

1992 

Fuzzy set theory 

/(A) = - (2A(*) - hi 

xeX 

(9.34) 

Fuzziness 

1979 

Fuzzified evidence theory 

F(m) = X>(A)/<>4) 

(9.61) 

Fuzziness 

1988 
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EXPERT OPINION ELICITATION 


The process of expert opinion elicitation can be used to deal with uncertainty and risk 
in cases were data and experiences are absent. 


Expert Opinion Elicitation 


> System Complexity 

> Delphi method 

s technological forecasting 
S policy analysis 

> Scenario analysis 

> The basic Delphi method by Helmer (1968) 

> Nuclear Regulatory Commission Method 
(1999) 
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EXPERT OPINION ELICITATION 


The process of expert opinion elicitation is described in steps. 


Expert Opinion Elicitation 

'■'V. .^mam M ■»».. ■■■■■■ ■HROH 

> Delphi method 

s technological forecasting 

S policy analysis 

> Scenario analysis 

> The basic Delphi method consists of the following 
steps (Helmer 1968): 

S Selection of issues or questions and development of 
questionnaires. 

■s Selection of experts who are most knowledgeable about 
issues or questions of concern. 

s Issue familiarization of experts by providing sufficient details 
on the issues on the questionnaires. 

OTSM 
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EXPERT OPINION ELICITATION 


The process of expert opinion elicitation is described in steps. 


Expert Opinion 

> The basic Delphi method (Cont.)T~ ” ' 

S Elicitation of experts about the issues. The experts generally do 
not know who the other respondents are. 

s Aggregation and presentation of results in the form of median 
values and an inter-quartile range (i.e., 25% and 75% percentile 
values). 

s Review of results by the experts and revision of initial answers by 
experts. Respondents who provide answers outside the inter- 
quartile range need to provide written justifications or arguments 
on the second cycle of completing the questionnaires. 

S Revision of results and re-review for another cycle. The process 
should be repeated until a complete consensus is achieved. 
Typically, the Delphi method requires two or three cycles or 
iterations. 

s A summary of the results is prepared with argument summary for 
out of inter-quartile range values. 
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Figure 24 
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EXPERT OPINION ELICITATION 


The process of expert opinion elicitation is described in a flowchart. 


Expert Opinion 
Elicitation 
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EXPERT OPINION ELICITATION 


The outcomes of expert opinion elicitation is described including consensus and no 
consensus. 



Figure 26 





SUMMARY AND CONCLUDING REMARKS 


A Hierarchy for ignorance is provided in this paper, and analytical methods are 
identified for modeling various types. Novel analytical methods and algorithms to accurately 
assess and model information content by classifying, analyzing and modeling ignorance 
types for the purpose of constructing knowledge are outlined in the paper. As our reliance on 
computational methods in simulation-based approaches for discovery and design, the need 
for formal methods to analyze and model ignorance and uncertainty is expected to increase. 
These methods can be used within a framework of decision analysis to meet the needs of 
decision and policy makers. 


Summary and 
Concluding Remarks 


> Characteristics of future systems: 

■s Complexity 
■s Uncertainty 
•s Societal expectation 
S New risks 
S Risk acceptance 

> Simulation and decision-based design requires 
quantitative methods for verification and 
validation. 


> 


or m 


The need to develop qualitative methods that 
are suitable for engineering systems to deal 
with uncertainty and ignorance in order to 
manage risk. 
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OUTLINE OF THE PRESENTATION 


The following references describe in more detail the material given in this 
presentation. These references are available from the appropriate sources or can be obtained 
from William Oberkampf: wloberk@sandia.gov . 

1. Oberkampf, W. L., Diegert, K. V., Alvin, K. F., and Rutherford, B. M., 
"Variability, Uncertainty, and Error in Computational Simulations," ASME-HTD-Vol. 357- 
2, AIAA/ASME Joint Thermophysics and Heat Transfer Conference, Albuquerque, NM, 
1998. 

2. Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and 
Alvin, K. F., "A New Methodology for the Estimation of Total Uncertainty in Computational 
Simulation," American Institute of Aeronautics and Astronautics, AIAA Paper No. 99-1612, 
AIAA/ASME/ ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference 
and Exhibit, St. Louis, MO, 1999. 

3. Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and 
Alvin, K. F., “Estimation of Total Uncertainty in Modeling and Simulation,” Sandia National 
Laboratories Report, SAND2000-0515, Albuquerque, NM, March 2000. 

4. Oberkampf, W. L., Helton, J. C., and Sentz, K. “Mathematical Representation 
of Uncertainty,” American Institute of Aeronautics and Astronautics, AIAA Paper No. 2001- 
1645, AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials 
Conference and Exhibit, Seattle, WA, 2001. 


Outline of Presentation 

• Background and purpose 

• Phases of nondeterministic simulation 

• Distinction between uncertainty and error 

• Application of the framework to a free-flight missile simulation 

• Summary and conclusions 
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BACKGROUND AND PURPOSE 


Our focus is on developing a framework for identifying and estimating error and 
uncertainty in nondeterministic computational simulation. This framework is composed of 
six phases, which represent a synthesis of the activities recognized in the systems engineering 
(operations research) community, the probabilistic risk assessment community, and the 
numerical methods community. Our framework emphasizes models that are given by a set of 
partial differential equations (PDEs) that must be solved numerically, although the 
framework is also applicable to modeling in general. We stress a clear distinction between 
the specification of the system, which is modeled by a set of PDEs, and the environment, 
which should be representative of the boundary conditions and excitation for the PDEs. We 
make a distinction between error and uncertainty so that the issues of representation and 
propagation of each is aided. The issue of numerical solution error is generally ignored in 
risk assessment analyses and nondeterministic simulations. Neglecting numerical solution 
error can be particularly detrimental to uncertainty estimation when the mathematical models 
of interest are cast in terms of nonlinear PDEs. Types of numerical error that are of concern 
in the numerical solution of PDEs are spatial discretization error in finite element and finite 
difference methods, temporal discretization error in time-dependent simulations, and error 
due to discrete representation of strongly nonlinear interactions. 


Background and Purpose 

Develop a general framework for estimating uncertainty and error 
in nondeterministic computational simulations 

• Scope of framework: 

• Continuum mechanics and energy transport 

• Mathematical models are given by a system of ordinary or 
partial differential equations 

• Differential equations are solved by discretization methods 
(finite element, finite difference, finite volume methods) 

• Approach represents a synthesis of methods from: 

• Systems Engineering (nuclear reactor risk assessment) 

• Statistics (probabilistic structural mechanics) 

• Numerical solution of PDEs (finite element methods) 

7/16/01 /homa/wloberk/com p. va lida tio representations, mi ne/UQ/O DU. N AS A. wo rkshop.5-01 




Sanffa 

National 

Laboratories 


Figure 2 


70 






STEPS IN QUANTIFYING UNCERTAINTY AND ERROR IN 
NONDETERMINISTIC SIMULATIONS 


One could identify four steps in quantifying uncertainty and error in nondeterministic 
simulations. First, one constructs a mathematical model of the system of interest. This 
model must define where the physical system ends and the environment, or surroundings, 
begin. Second, one must specify where and how all of the modeled uncertainties and errors 
appear in the formulation of the nondeterministic simulation. Third, given these uncertainties 
and errors, one assumes a mathematical representation that will be used to describe these 
uncertainties and errors. For example, uncertainties are traditionally represented by 
probability distributions. Fourth, one must propagate and aggregate these mathematical 
representations of uncertainty and error through the nondeterministic computational process. 


Steps in Quantifying Uncertainty and Error 
in Nondeterministic Simulations 



Sanifia 

National 

Laboratories 


Engineering sciences Center 


1 ) Construct a mathematical model of the system of interest 


2) Identify all relevant sources of uncertainty and error 


3) Create appropriate mathematical representation for each 
individual source of uncertainty and error 


4) Propagate and aggregate all representations of sources through 
the nondeterministic simulation process 


7/16/01 


/home/wloberk/co mp.valida bon/prasentations. mine/UQ/OD U. NAS A. workshop. 5-0 1 


Figure 3 


71 






PHASES IN NONDETERMINISTIC SIMULATIONS 


This presentation proposes a comprehensive new framework, or structure, of the 
general phases of modeling and simulation. The phases are, 1) conceptual modeling of the 
physical system, 2) mathematical modeling of the conceptual model, 3) discretization and 
algorithm selection for the mathematical model, 4) computer programming of the discrete 
model, 5) numerical solution of the computer program model, and 6) representation of the 
numerical solution. Characteristics and activities of each of the phases are applicable to a 
variety of disciplines, e.g., computational fluid dynamics, structural dynamics, and heat 
transfer. We also distinguish between aleatory uncertainty, epistemic uncertainty, and error 
that might occur in any of the phases of modeling and simulation. 


Phases in Nondeterministic Simulations 
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TYPES OF UNCERTAINTY 


We use the term aleatory uncertainty to describe the inherent variation associated 
with the physical system or the environment under consideration. Sources of aleatory 
uncertainty can commonly be singled out from other contributors to nondetermini Stic 
simulation by their representation as distributed quantities that can take on values in an 
established or known range, but for which the exact value will vary by chance from unit to 
unit or from time to time. Aleatory uncertainty is also referred to in the literature as 
stochastic uncertainty, variability, inherent uncertainty, and irreducible uncertainty. We 
define epistemic uncertainty as a potential deficiency in any phase or activity of the modeling 
process that is due to lack of knowledge. The first feature that our definition stresses is 
"potential," meaning that the deficiency may or may not exist. In other words, there may be 
no deficiency, say in the prediction of some event, even though there is a lack of knowledge 
if we happen to model the phenomena correctly. The second key feature of epistemic 
uncertainty is that its fundamental cause is incomplete information. Incomplete information 
can be caused by vagueness, non-specificity, or dissonance. Epistemic uncertainty is also 
referred to as reducible uncertainty and ignorance. 


Types of Uncertainty 

Aleatory uncertainty is the inherent variation associated with the 
physical system or the environment. 

• Also referred to as irreducible uncertainty, variability, and 
stochastic uncertainty. 

• Examples: 

• Variation in thermodynamic properties due to manufacturing 

• Variation in joint stiffness and damping in structures 

• Variation in external excitation of a system 

Epistemic uncertainty is a potential deficiency in any phase of the 
modeling process that is due to lack of knowledge. 

• Also referred to reducible uncertainty, model form uncertainty, 
and subjective uncertainty. 

• Examples: 

• Poor understanding of fracture dynamics 

• Poor knowledge of failure, misuse, or hostile scenarios 

• Information from expert-opinion elicitation 
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ERROR IN COMPUTATIONAL SIMULATIONS 


We define error as a recognizable deficiency in any phase or activity of modeling and 
simulation that is not due to lack of knowledge. Our definition stresses the feature that the 
deficiency is identifiable or knowable upon examination; that is, the deficiency is not caused 
by lack of knowledge. Essentially there is an agreed-upon approach or ideal condition that is 
considered to be more accurate. If divergence from the correct or more accurate approach is 
pointed out, the divergence is either corrected or allowed to remain. It may be allowed to 
remain because of practical constraints, such as the error is acceptable given the 
requirements, or the cost to correct it is excessive. This implies a segregation of error types: 
an error can be either acknowledged or unacknowledged. Acknowledged errors are those 
deficiencies that are recognized by the analysts. When acknowledged errors are introduced 
by the analyst into the modeling or simulation process, the analyst typically has some idea of 
the magnitude or impact of such errors. Examples of acknowledged errors are finite 
precision arithmetic in a computer, approximations made to simplify the modeling of a 
physical process, and conversion of PDEs into discrete equations. Unacknowledged errors 
are those deficiencies that are not recognized by the analyst, but they are recognizable. 
Examples of unacknowledged errors are blunders or mistakes; that is, the analyst intended to 
do one thing in the modeling and simulation but, for example, as a result of human error, did 
another. There are no straightforward methods for estimating, bounding, or ordering the 
contribution of unacknowledged errors. 


Error in Computation Simulations 

Error is recognizable deficiency in any phase of the modeling and 
simulation process that is nof due to lack of knowledge. 

• Acknowledged errors are errors that can be estimated, bounded, or 
ordered 

• Finite precision arithmetic in a digital computer 

• Lack of spatial grid convergence 

• Conversion from continuum PDEs to discrete mathematics 

• Unacknowledged errors are blunders or mistakes: 

• Programming errors 

• Input and output errors 

• Compilation and linkage errors 
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EXAMPLE PROBLEM: MISSILE FLIGHT ANALYSIS 


In this example we consider an analysis of the flight of a rocket-boosted, aircraft- 
launched missile. We make the following assumptions concerning the missile: 

1. The missile is unguided during its entire flight, i.e., only ballistic flight is 
considered. 

2. The missile is propelled by a solid fuel rocket motor for the initial portion of 
its flight, and it is unpowered during the remainder of the flight. 

3. The missile is fired from a launch rail attached to the aircraft in flight. 

4. The only aerodynamic surfaces on the missile are fins to provide flight 
stability. 

The analysis considers the missile flight to be in the unspecified future, i.e., the 
analysis is an attempt to predict future plausible events, not analyze an event in the past. The 
analysis requires the estimated uncertainty in all of the plausible events. 


Example Problem: /JS \ Santfia 

Missile Flight Analysis l!l!l Laboratories 

• Problem description: 

• Rocket-boosted, aircraft-launched missile 

• Unguided during entire flight 

• Propelled by a solid fuel rocket motor 

• Fired from a launch rail on the aircraft 

• Typical purposes of nondeterministic analyses: 

• 1) Missile performance (normal environments) 

• 2) Flight safety (abnormal environments) 

• 3) Missile reliability (hostile environments) 
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PHASES OF NONDETERMINISTIC SIMULATIONS 


We have identified four major activities that are conducted in the conceptual 
modeling phase: system/environment specification, scenario abstraction, coupled physics 
specification, and nondeterministic specification. The system/environment-specification 
activity consists primarily of carefully identifying the physical or conceptual elements that 
are considered part of the system and those that are considered part of the environment. The 
scenario-abstraction activity attempts to identify all possible physical events, or sequences of 
events, that may affect the goals of the analysis. The coupled physics specification identifies 
and carefully distinguishes the possible alternatives for physical and chemical processes in 
the system, and the coupling between these processes for the system/environment 
specification and scenario abstraction under consideration. In the nondeterministic 
specification activity, decisions are made concerning what aspects of the system and 
environment will be considered deterministic or nondeterministic. 

We have identified four major activities in the mathematical modeling phase: 
formulation of the PDEs, selection of all auxiliary equations that supplement the differential 
equations, formulation of all initial and boundary conditions required to solve the PDEs, and 
selection of the mathematical representation of nondeterministic elements of the analysis. 
The PDEs commonly represent conservation equations for mass, momentum, and energy, but 
they can originate from any mathematical model of the system. The auxiliary equations are 
equations that are required to complete the PDEs. The boundary and initial conditions 
provide the required closure equations needed for all PDEs. Formulation of the 
nondeterministic representations is based on the needs of the analysis, as well as the quantity 
and quality of relevant information available. 


Phases of Nondeterministic Simulations 
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CONCEPTUAL MODELING ACTIVITIES 


Three possible system/environment specifications for the missile flight example are 
shown. The specifications are listed from the most inclusive (with regard to the system 
specification) to the least inclusive. System/Environment Specification 1 considers the 
missile and the atmosphere near the missile to be part of the system, whereas the launching 
aircraft and target are considered part of the environment. System/Environment 
Specification 2 considers the missile and the aerothermal processes occurring on the missile 
to be part of the system, whereas the atmosphere near the missile, the launching aircraft, and 
the target are considered part of the environment. System/Environment Specification 3 
considers the missile to be the system, whereas the aero thermal processes, atmosphere near 
the missile, launching aircraft, and target are considered part of the environment. Even 
though this is the simplest specification considered, it still allows for significant complexities 
in the analysis. Note that in the diagram the only specification, or tree element, delineated is 
System/Environment Specification 3. 
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ACTIVITIES IN THE REMAINING PHASES OF NONDETERMINISTIC 

SIMULATIONS 


The discretization and algorithm selection phase accomplishes two related activities. 
First, it converts the continuum mathematics model, i.e., the differential equations, into a 
discrete mathematics problem suitable for numerical solution. Second, it provides the 
methodology determining how a discrete set of computer solutions can be most appropriately 
used to accommodate the nondeterministic features of the analysis. Three activities are 
identified in the computer-programming phase: input preparation, module design and coding, 
and compilation and linkage. Input preparation refers to the analyst's conversion of the 
mathematical and discrete model elements into equivalent data elements usable by the 
application code. The second and third activities relate to the building of the application 
code itself. Four activities are identified in the numerical solution phase: spatial and 
temporal convergence, iterative convergence, nondeterministic propagation convergence, and 
computer round-off accumulation. In the solution representation phase we have identified 
five activities: input preparation, module design and coding, compilation and linkage, data 
representation, and data interpretation. The first three activities are very similar to those 
discussed in the computer-programming phase. The data representation activity includes two 
types of similar activities: 1) the representation of individual solutions over the independent 
variables of the PDEs and 2) a summary representation that combines elements of the 
multiple individual deterministic computer runs. The data interpretation activity refers to the 
human perceptions or impressions that are formed based on observation of the represented 
solutions. 


Activities in the Remaining 

Phases of Nondeterministic Simulations 
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Computer Programming 
Activities 



Input Preparation 
(Unacknowledged Errors) 

Module Design and Coding 


(Unacknowledged Errors) 

Compilation and Linkage 
(Unacknowledged Errors) 


Discretization and 
Algorithm Selection 
Activities 


Discretization of PDEs 
(Acknowledged Errors) 
Discretization of BCs and ICs 
(Acknowledged Errors) 


Selection of Propagation Methods 
(Acknowledged Errors) 


Design of Computer Experiments 
(Acknowledged Errors) 


Solution Representation 

Numerical Solution Activities 
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TREE-STRUCTURE FOR MODELS, SOLUTIONS, AND REPRESENTATIONS 


The figure illustrates the multiple models, numerical solutions, and solution 
representations that are addressed in the missile flight example. As shown in the figure, six 
conceptual models are identified, many more are implied, but for illustration only one is 
selected for further development and analysis. This single conceptual model spawns two 
alternative mathematical descriptions, the 3-DOF and 6-DOF models, both of which are 
carried through the remaining phases of the modeling and simulation process. For simplicity, 
only one of these mathematical models shows further development, although it is understood 
that identical development of Mathematical Model 1 is taking place in parallel with 
Mathematical Model 2. The discretization and programming phases identify alternative 
model choices that are not considered further in this example. Continuing into the numerical 
solution phase, nondeterministic effects that were identified in the conceptual model and 
further defined in the mathematical modeling phase are computed via multiple deterministic 
numerical solutions. Flow these solutions were computed was specified in the propagation 
method identified in the discretization and algorithm selection phase. Finally, in the solution 
representation phase, the multiple solutions are merged to represent the complete 
nondeterministic solution. 



Figure 1 1 
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SUMMARY AND CONCLUSIONS 


We have presented a comprehensive, new framework for modeling and simulation 
that blends the perspective of three technical communities: the systems view from the 
operations research community, propagation of uncertainty from the risk assessment 
community, and the numerical solution of PDEs from the computational physics community. 
The activities that are conducted in each of the six phases of modeling and simulation were 
discussed. We carefully define and distinguish between uncertainty and error. Our 
framework applies regardless of whether the discretization procedure for solving the PDEs is 
based on finite elements, finite volumes, or finite differences. The formal distinction 
between aleatory uncertainty and epistemic uncertainty in this framework drives one toward 
different mathematical representations for each type of uncertainty. Probabilistic 
representations are clearly appropriate for aleatory uncertainty, and various other modem 
information theories are, we believe, more appropriate for epistemic uncertainty. We 
recommend research into evidence (Dempster/Shafer) theory. This theory, however, is not 
well developed when compared to traditional probabilistic methods. If one were to take the 
step and represent aleatory uncertainty probabilistically and epistemic uncertainty with 
evidence theory, then one must face the question of propagating these components 
concurrently through the modeling and simulation process. Propagation of Belief and 
Plausibility measures from evidence theory through complex PDE models is a research topic. 


Summary and Conclusions 
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• Presented an overview of the phases of nondeterministic 
simulation and the activities that occur in each phase 

• Nondeterministic analyses should be focused, e.g., performance, 
reliability, or risk assessment. 

• We have distinguished between sources of aleatory uncertainty, 
epistemic uncertainty, and error 

• Mathematical representations of uncertainty: 

• Aleatory uncertainty: traditional probability theory 

• Epistemic uncertainty: Dempster-Shafer theory 

• Areas of research in Dempster-Shafer theory: 

• Construction of input Belief and Plausibility measures 

• Combination of evidence is non-unique 

• Propagation of Belief and Plausibility measures through the model 
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Errors and Uncertainties in Probabilistic Engineering Analysis 
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Motivation 


• Confidence in probabilistic assessments is required to 
support 

* Certification 

* Design analysis 

* Critical decisions 

» First step towards verification and validation of probabilistic 
models 

» Understanding where errors originate wilt suggest more 
robust computational strategies 
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Sources of Error 

in Probabilistic Analysis 

* Model approximation 


* First or second-order approximation 

• Calculation of derivatives 


* Uncertainty characterization 


* Insufficient data 

■ AH forms of error are reducible 

» Selection of incorrect distribution 

■ V&V of the probabilistic analysis 

* Probability integration 

■ Increased data collection 

* Insufficient number of samples 

■ Development of more accurate and 

• First or second-order approximation 

robust analysis methods 

■ Numerical algorithm 


« Transformations to standard normal 


• Convergence error in finding the MPP 
« Algorithm error (wrong or multiple MPP) 
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Algorithm Error is Most Troublesome 


■ Source of error is inability of the 
algorithm to locate the correct MPP 

* focal minimum 

* multiple minimums 

* violations of the assumptions of: a smooth 
and continuous response surface. 

* For robustness, algorithm must be 
able to locate all MPP's 

* Troublesome aspect is that problem 
can arise after transformation to 
standard normal, unbeknownst to 
the user 

* Can occur when mapping from 
original to standard normal space 
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Probabilistic Analysis Methods 


« Fast Probability Integration Methods 

* Advanced mean value 

* First and second-order reliability methods 
« Fast convolution integration using FFT 

* Sampling Methods jM 

« Monte carlo simulation 

* Sphere-based importance sampling 

* Latin hypercube simulation 

* Adaptive importance sampling 

■ Probabilistic fault-tree 

* Response surface method 




Most Probable 
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Convergence Criteria 
"Modified RF" Optimization 
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Example Problems 

■ Three examples solved 

» From SAE G1 1 Probabilistic Methods, Numerical Review Subcommittee 

« Goal is to investigate accuracy and error (not efficiency) 

* Solutions obtained using 

* Monte Carlo (MCS) 

* First-order Reliability Method (FORM) 

* Second-order Reliability Method (SORM) 

* Advanced Mean Value (AMV+) 

* Adaptive importance Sampling (AIS) 

* Response Surface Method (RSM) 

• NESSUS 3,0 software used for all problems 

* Modified Rackwitz-Fiessler (RF) attempted first 

* If problems, switched to sequential quadratic programming (SQP) 


•iw 
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Example 1 - Gear Contact Stress Model 


Response given by 

•22*©? $in # 
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Example 1 - Gear Contact Stress Model 


■ $f methods 
compare 



reasonably well 




■ RSM in error in 
left tail 

* Due to fit around 
centre) region 
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■ But what is the 
error? 
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Example 1 - Gear Contact Stress Model 


■ Percent error from 
Monte Carlo 
solution (100M 
samples) 

* 95% confidence 
bounds 

■ All methods posses 
some error 

» Xrspace and u-space 
function approximation 
« convergence tolerance 



Even for mildly nonlinear problems, accurate function 
approximations in x and u-space are important 


l-Smthw 


Figure 15 


Example 2 - Maximum Radial Stress of a 
Rotating Disk 


» Response function 

K) £■ 

X \ 8 J V ( 9 . 8 1 )( 39 . 37 ) /V 60 j y ' ' 1 

■ Random Variables 


Var 

Name 

Mean 

SD 

Dist 

V 

Poisson's 

Ratio 

0.30 

0.005 

Normal 

p 

Density 

(Ib/in3) 

0.284 

0.002 

Normal 

CO 

Rotor 

Speed 

(rpm) 

10500 

288.7 

Uniform 

(10000,11000) 

ro 

Outer 
Radius (in) 

8 

002 

Normal 

ri 

Inner 

Radius (in) 

2 

0.01 

Normal 
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Figure 17 


Example 2 - Maximum Radial Stress of a 
Rotating Disk 


■ Plot shows g=0 contours 
for all 9 levels 

» Modified RF converged 
on only 4 MPRfe 

» Switched to SQP and 
found all MPP's 

« Note that first-order fit to 
response reasonably 
good near origin, and 
worse in tails 
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Figure 19 



Figure 20 
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Example 2 - Maximum Radial Stress of a 
Rotating Disk 


Lessons Learned 


* First-order approximation in u -space may not be accurate for 
highly nonlinear function 

• Nonlinear in x-space 

* Nonnormal transformation 

* AMV+ converges on location of MRP. Computed probability 
then depends on what order of approximation is used. 

* AMV+ used here is based on first-order. Second order 
available, but not used very often. 

* Recommendation is to use ASVIV+ to locate MPP. Then use 
AfS to compute probability. 
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Figure 23 


Example 3 - Multiple MPP Problem 


■ g=0 levels numbered in 
contour plot 

■ Again, the modified RF 
algorithm reported 
convergence errors. 

■ Switched to SQP and found 
one MPP per limit state 

■ Function is symmetric about 
X2 - therefore, two MPP's 
exist. 

■ Solutions reported on 
following charts are based on 
one (shown) MPP 



24 


Figure 24 
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Example 3 - Multiple MPP Problem 


■ Very steep cdf in 
left tail 

■ Unable to discern 
difference in 
solutions 

* Systematic error in 
right tail observed ’ 

■ RSM obviously bad 

* Full quadratic is poor 
approximation to 
original function 

* Using wider move limits 

improved tait, but at *0 ««i : s s» cm m m 1150 

expanse of central * 

region 

■South west Research Institute 
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Example 3 - Multiple MPP Problem 


■ Left tail response 
shown 

■ Error observed at 
levels 1-6 by all 
methods 

« Error due to use of 
only one MPP 

■ R$M error due to 
poor function 
approximation 

-zo sxj Qf>. 


Sovfowe s? Research institute 

Figure 26 
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Example 3 - Multiple MPP Problem 


* Percent error 

■ All methods have 
significant error 

■ Need a more genera! > 
approach that can 
automatically 

* switch optimizers if trouble 
encountered 

« identify multiple MPP's 

* solve multiple MPP 
problem 
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Proposed Strategy: System Approach 


* Case 5, Level 4 studied in 
more detail \ 

•02 

4- 
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1 

* Monte Carlo sampling on \ 

transformed response function 1 
(u-space) 1 

M 
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Proposed Strategy: System Approach 


■ Use NESSUS probabilistic 
fault tree to solve multiple 
MPP problem 

■Adaptive Importance 
Sampling used to compute 
union of two limit states 

■ Sampling based MPP search 
and system probability 
calculation applied to each 
level 

■ Excellent results obtained 


Figure 29 


Summary 

■ All forms of error are reducible 

* V&V of the probabilistic analysis 
m Increased data collection 

* Development of more accurate and robust analysis methods 
» Assessed via deterministic or probabilistic analysis 

* Types of errors in probabilistic analysis 

* Model approximation 

* Uncertainty characterization 
« Probability integration 

* Numerical algorithm 

* Various errors quantified in paper using simple analytical 
models 

» Simple but effective strategy used to solve multiple MPP 
problem 

> lj| 3l Sotilhvms! R&searsif Institute 

Figure 30 
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Conclusions 


* For challenging, problems, trained analysts are required to 
compute accurate solutions 

■ Advanced mean value (AMV+) method and the modified RF 
optimizer were the only methods that identified when 
problems were occurring 

■ Solutions to difficult problems (multiple MPP) can be solved 
using existing methods, but detailed knowledge of the 
problem is required 

■ Robustness of computer code not only measured by its 
ability to get a solution, but to get the right solution 

» Solution error must be quantified before confidence is gained 

# Methods must warn when problems are suspected 

* Adaptive/intelligent methods can produce accurate solutions 

SoiJlhwesi Research tostikM# 


Figure 3 1 
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OUTLINE 


Existing predictive technologies 
Probabilistic technology 
Practical software architecture 
Typical Inputs 
Typical outputs 
Examples 

Latest development in probabilistic software tools 



■ Existing predictive technologies 

■ Probabilistic technology 

■ Practical software architecture 

■ Typical Inputs 

■ Typical outputs 

■ Example 

■ Latest developments in probabilistic software tools 


Unipass Technologies 


Figure 1 
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EXISTING PREDICTIVE TECHNOLOGIES 


Traditionally, predictive analyses have used either a deterministic or a statistical 
approach. The deterministic approach attempts to predict the outcome of a process or event 
using physics-based mathematical models of the event by assigning single values to the 
variables that enter these models and affect the outcome. Furthermore, the deterministic 
approach also assign a safety factor to the outcome in order to account for the overlooked 
variability and uncertainties. 

In contrast, the statistical approach relies on pure statistical data of process or event 
out come that can often be flawed or difficult to obtain. Both of these approaches work in 
isolation, failing to consider any other factors that may significantly affect the result. 


'^Exisl^a ^^edictive Technologies Have Many Shortcomings 

■ Deterministic approaches 

• Develop a deterministic event model using physics, process, rules, etc 

• Assign a single value to each deterministic event model variable and calculate a 
single value for the outcome 

• Calculate a single adjusted value of the outcome by applying a safety factor to the 
calculated outcome to compensate for uncertainties 

• Predict future of the event using the single adjusted value of the outcome 

• Confirm and/or calibrate the single adjusted value by tests and/or field data 

■ Statistical approaches 

• Collect sufficient statistical data for the similar event outcome 

• Calculate mean, variance, and distribution of the the event outcome using 
collected data 

• Predict future of the event using the mean, variance, and distribution of the the 
event outcome 


Unipass Technologies 


Figure 2 
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PROBABILISTIC TECHNOLOGY 


The probabilistic analysis begins by developing the deterministic models of the 
process and identifying the uncertainties associated with such process. There are two type of 
uncertainties that should be identified, namely, Type I Uncertainties and Type II 
Uncertainties. The Type I Uncertainty representing the inherent uncertainties associated with 
the deterministic model variables (e.g., variability of the load in a stress model that calculates 
the stress under the effect of such load) while the Type II Uncertainties are the uncertainties 
that are not covered under the Type I Uncertainties (e.g., uncertainties associated with lack of 
statistical data, model imperfection, human error, measurement error, etc.). If necessary, new 
variables should be defined to represent Type I and II Uncertainties. Upon completion of the 
deterministic models and identification of the uncertainties, the Probabilistic Process Models 
(PPM) should be built. These models describe the outcome of an event using the 
deterministic models and the Uncertainty Type I and 13 variables. The Variable Probability 
Distribution Models (VPDM) is built using the available test/field data and/or the analyst 
judgment. The PPM and VPDM are then used to perform probabilistic analysis utilizing a 
probabilistic software engine (e.g., UNIPASS™). 


Probabilistic Technology Eliminates 
The Shortcomings of The Existing Technologies 

■ Probabilistic approaches 

• Develop a deterministic process models using physics, process, rules, etc 

• Identify uncertainties associated with the deterministic-process-models variables 
( Type I Uncertainties) 

• Identify uncertainties that are not associated with the deterministic- model variables 
( Type II Uncertainties) 

• Define new variables representing Type I Uncertainties if necessary 
(Type I Uncertainty Variables) 

• Define additional variables representing Type II uncertainties 
(Type II Uncertainty Variables) 

• Develop a probabilistic -process-model by incorporating all Type I and II Uncertainty 
Variables into deterministic event model 

• Develop variable statistical models for all variables including deterministic model 
variables, Type I Uncertainty Variables, and Type II uncertainty Variables using test/field 
data and/or analyst judgment 

• Perform probabilistic analysis using probabilistic event model and statistical variable 
models 

• Predict future of the event using the obtained results 

• Calibrate/update models using tests and/or field data 

Unipass Technologies 


Figure 3 
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PROBABILISTIC TECHNOLOGY 


The probabilistic approach incorporates the best of deterministic and statistical 
methods. This is achieved by using the deterministic process models and enhancing them by 
taking into account the uncertainties associated with the process and variables (e.g., inherent 
uncertainties, modeling and measurement errors, lack of data, etc.) and identifying the 
individual factors that are key to predicting a likely outcome. This approach optimizes the 
analysis, zeroing in on the factors that actually drive the process. 

As oppose to statistical approach the probabilistic technology utilizes the information 
regarding the variables that enter the process models and affect the outcome and does not 
require process statistical data for the analysis, however, such data may be used to fine tone 
the predictive models. Furthermore, the technology quantifies the safety measures and 
prediction accuracy by providing probabilities associated with the process outcomes. 



Probabilistic Technology 
Is Superior to The Existing Predictive Technologies 


Predictive Technology Comparison Chart 


Utilizes physics/behavioral/rule/process based predictive model 

1 

X 

■ 

Considers inherent uncertainties, modeling uncertainties, lack 
of data, human error, measurement error 

X 

X 

— 

Compensates for unknowns using: 

SF 

SM 

SM 

Utilizes past performance data to improve accuracy 

/ 


\ /. 

Does not require event’s past performance data to develop 
predictive model 

j| 

xv 

Quantifies safety measures 

u u 

• / 

Quantifies prediction accuracy 

u u 

T ' / 

\ f 

[71 TRUE £3 FALSE JJJUNKNOWN [sFj SAFETY FACTORS [^STATISTICAL METHODS 
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PRACTICAL SOFTWARE ARCHITECTURE 


A practical software architect works through the combination of one or more 
databases containing the information about the process variables and/or outcome; a Filter 
software that cleans up the databases using a series of rules identified by the analyst; a 
software engine that reads the data from the databases and identifies the best Variable 
Probability Distribution Models (VPDM) for the process variables (e.g., our ProFit™ 
software); a software engine that can create the Probabilistic Process Model (PPM) using the 
deterministic models, from the deterministic software systems, and the information residing 
in the databases (e.g., our ProModeler™ software); and probabilistic engine (e.g., our 
UNIPASS™ software engine) that perform the probabilistic analysis. The PPM is a 
mathematical representation of how an event works which may be behavior, process, physics 
and/or rule based. The VPDM is the probability distribution of the PPM’s variables. 





Practical Software Architecture 
Utilizes Many Many Software Tools and Databases 


Update Variable Models 



Unipass Technologies 


Figure 5 
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TYPICAL INPUTS 


The probabilistic analysis requires four essential items including (a) the Probabilistic 
Process Models (PPM) that describe the outcome of an event and are constructed using the 
deterministic models (e.g., a finite element model or a crack growth model) and the 
mathematical models that describe the uncertainties and are not considered in the 
deterministic models (e.g., lack of statistical data, modeling error, human error, measurement 
error, etc.); (b) the PPM’s Deterministic Variables; (c) the probability distribution models 
that describe the randomness of the Uncertainty Type I Variables (e.g., Type I Uncertainties); 
and (d) the probability distribution models that represent the randomness of the Uncertainty 
Type II Variables (e.g., Type II Uncertainties). 



Typical Inputs Provides for 
Integration of Process and Uncertainties 


Informed 

Decision 



Unipass Technologies 


Figure 6 
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TYPICAL INPUTS - DEFINING VARIABLE PROBABILITY 
DISTRIBUTION MODELS 


Large Amount of Data Available: One of the following techniques may be use to 
construct the Variable Probability Distribution Models (VPDM) when large amount of data is 
available (more than 50 data point). 

Approach 1: Assume a distribution type and calculate the distribution parameters 
using the method of moments. Perform the goodness-of-fit tests to eliminate the 
unacceptable distributions. Use probability paper approach to select the best distribution in 
the desired range. 

Approach 1: Assume a distribution type and calculate the distribution parameters 
using the maximum likelihood method. Perform the goodness-of-fit tests to eliminate the 
unacceptable distributions. Use probability paper approach to select the best distribution in 
the desired range. 


Variable Models 


■ Large amount of data available 

• Distribution type: identify distribution type using probability paper 
and/or goodness-of-fit-tests. Must have large amount of data 
(usually 50-100 points) to discriminate between distribution types. 
Common tests are K-S test and anderson-darling test (preferred). 
Identify distribution type using probability paper and/or goodness- 
of-fit-tests 

• Distribution’s parameters: calculate distribution’s parameters using 
method of moments or maximum likelihood estimator for the 
selected distribution type 


Unipass Technologies 
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TYPICAL INPUTS - DEFINING VARIABLE PROBABILITY 
DISTRIBUTION MODELS 


Small Amount of Data Available: Assume a distribution type and calculate the 
distribution parameters using the maximum likelihood method. Utilize 
hyperparameterization (assume parameters of the random variables to be random variable 
themselves) to account for lack of sufficient data. Use probability paper and/or the 
goodness-of-fit test approach to identify all acceptable distribution types within the desired 
range. Select the distribution that provides for the most conservatism. 



■ Small amount of data available 

• Distribution’s parameters: Calculate distribution’s parameters using 
maximum likelihood estimator for the selected distribution type and use 
hyperparameterization to consider uncertainty due to lack of statistical 
data 


• Distribution Type: Select a distribution that is conservatively consistent 
with the available data 
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TYPICAL INPUTS - DEFINING VARIABLE PROBABILITY 
DISTRIBUTION MODELS 


Very Limited Amount of Data Available: Assume Beta or Uniform distribution type. 
Use hyperparameterization for poorly quantified physical bounds. 

No Data Available But Engineering Analysis Exists: Construct a user-defined 

distribution as shown in the example below. 


Variable Models 

■ Very limited amount of data available 

• Use Uniform or Beta (good quantification of physical bound) 

• Use Uniform or Beta with Hyperparameterization (poor quantification of 
physical bound) 

■ No data available but engineering analysis exists 

• User defined distribution 

• Example 

> Worst stress value from FE analysis 120 ksi 

> Maximum spread between best and worst condition is 20 ksi 

> Actual stress could be off by 10% 

> Stress = C (120-S) 

Where C is uniform between 0.9 and 1.1 and S is uniform between 0 
and 20 

Unipass Technologies 


Typical Inputs Include 


Figure 9 
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TYPICAL INPUTS - DEFINING VARIABLE PROBABILITY 
DISTRIBUTION MODELS 


No Data or Engineering Analysis Exists: Construct a user-defined distribution as 
shown in the example below. 


Variable Models 

■ No data or engineering analysis available only engineering 
judgment can be made 

• User defied distribution 

• Example: consider a turbine blade thickness 

> Nominal value=0.058 

> Allowable tolerance = ±0.005 which occurs 90% of the time 

> Maximum deviation outside acceptable tolerance= ±0.002 (10% 
of the time) 



90% 


1 5% 


5% | 


0.051 0.053 0.063 0.065 
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Figure 10 
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TYPICAL INPUTS - DEFINING PROBABILISTIC PROCESS MODELS 


In general, an event model is a mathematical expression relating a dependent variable 
y, to a set of observable variables.*, and a set of unobservable model parameters z. The model 
is usually constructed on the basis of simplifying principals, and sometimes on purely 
heuristic basis when underlying phenomenon is not well understood. In addition to inherent 
uncertainties, there are at least three major phenomena that give rise to uncertainty in an 
event model. These include model imperfection, measurement error, and statistical 
uncertainty. The process of model construction mounts to estimating the unobservable model 

parameters z based on a set of measurements *j, and yj, i=l, m of the observable model 

variables. Following the Bayesian paradigm, we express our lack of precise knowledge about 
z by assigning a probability distribution to it. The Bayesian Updating Rule allows us to 
combine previous information about z with information obtained from observed data to arrive 
at a distribution. 



gUfes^’® 


Typical Inputs Include Process Models 


■ Known relationship (explicit or implicit) 

• In many instances, the exact theoretical relationship for the predictive 
model will be well known from a theoretical relationship such as from 
an engineering handbook or a deterministic code 

■ Know data to estimate predictive model 

• In some instances, it may be possible to construct an approximation to 
the predictive model by using regression models of empirical data or 
similar statistical methods 


Unipass Technologies 


Figure 1 1 
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TYPICAL OUTPUTS 


In general, probabilistic analysis produces three categories of information that would 
help achieving an informed decision. This includes probability information, most likely 
conditions, and sensitivity data. These categories are briefly descried in the following charts. 



Typical Outputs Provide for Predicting 
the Future, Understanding the Present, and Explaining the Past 


Unipass Technologies 



TYPICAL OUTPUTS 


Probability Information: The probability information includes reliably, failure 

probability, cumulative distribution function (CDF), and probability density function (PDF). 
The probability information may be used to identify the most critical failure mode; estimate 
the reliability, potential risk, and liability. It may also be used to determine the acceptable 
response range or calibrate the safety factors. 

Most Likely Conditions: The most likely conditions is identified by the most- 

probable-point (MPP). In general, the MPP represents the most likely values of the random 
variables at which the critical or significant condition of the user-defined event will occur. In 
engineering, a critical condition may be an undesirable event such as component failure or 
instability, or a desirable event such as extended component life or mission success. 


Output Includes Probability Information, 

Most Likely Conditions, and Sensitivity Data 

Probability information (e.g., failure probability, reliability, probability values, 
CDF, or PDF) can be used to: 

• Estimate reliability, failure probability, risk, and liability 

• Calibrate safety factor and identify critical failure mode 

• Minimize number of tests, inspection costs 

• Estimate response range 

• Etc 

Most-likely conditions (e.g., most-probable-point) can be used to : 

• Identify most-likely combination of predictive model variables in the field 

• Certification process tests, reliability demonstration tests, most likely test setups, 
safety control systems 

• Etc 


Unipass Technologies 


Figure 13 
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TYPICAL OUTPUTS 


In performing probabilistic analysis, it is often of interest to determine the sensitivity 
of event outcome or its calculated probabilities with respect to model variables and their 
parameters appearing in the event model. These measures are useful for many purposes 
including but not limited to identification of key event drivers, important sources of 
uncertainty, optimal condition, resource allocation, and analysis of model uncertainties. 
Often, these measures also provide insight into the physics of the event. 





Typical Outputs Includes Probability Information, 
Most Likely Conditions, and Sensitivity Data 


■ Sensitivity information includes physical and probability sensitivities 
and can be used to: 

• Identify key variables 

• Identify worst load combination 

• Automate processes 

• Minimize number of tests 

• Minimize weight 

• Minimize response variation 

• Minimize number of tight tolerances 

• Minimizes costs (e.G., Development, manufacturing, inspection, 
maintenance, and/or warranty costs) 

• Etc 
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EXAMPLE 


The reliability of the Design 2 should be equal or higher than the reliability of Design 
1. Investigate the possibility of using the less expensive material (Material 2) using 
deterministic approach. Check the accuracy of the results using probabilistic analysis. 



Example 


■ Select a material for design 2 that ensures equal or higher reliability 
than design 1 under the new loading condition 


Design 1: Load = P p Material = Material 1 


"Pi 

• Design 2: Loads = P, - P 2 , Material = ? 

p,- p 2 -h P, - P 2 


A, 

a 2 
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EXAMPLE 


Assuming a safety factor of 1.4 the deterministic analysis predicts a Safety Margin of 
0.0 for Design 1 and a Safety Margin of 0.131 for Design 2 using the Material 2. 

Analysis Results: Material 2 can be used for Design 2. 



Example 


■ Deterministic analysis strategy 

• Assume a safety factor of 1 .4 for both designs 

• Identify the margin of safety for both designs 

• Identify the safer design 

■ Deterministic analysis results 

• Design 1 : margin of safety = 1 40/(1 .4*1 00) - 1 = 0 

• Design 2: margin of safety = 95/(60*1 .4) - 1 = 0.1 31 

• Safer design: design 2 (0.131 > 0) 

• Sensitivity analysis results not available 
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EXAMPLE 


Assuming Material 2 for Design 2, probabilistic analysis predicts much higher failure 
probability for design 2 compared with Design 1. This violates the design requirement. 

Analysis Results: Material 2 cannot be used for Design 2. 



117 



EXAMPLE 


Design 2 is subjected to two random loads with 10% coefficient of variation. This 
will result in a standard deviation of 10.77 kips for the resultant load spectrum of Design 2. 
To ensure the same level of reliability for both designs, the number of standard deviation 
away from mean of the applied load must be identical for both designs. This assumption will 
result in a safety factor of 1.718 for Design 2. Using the new safety factor, the deterministic 
approach will also achieve the same results obtained by the probabilistic method. 



Example 


■ Calculate number of standard deviations away from mean (SD X ) for design 1 
based on assumed safety factor (SF, = 1.4) 

SDj = (SFj*100-100)/10 

= (1.4*100-100)/10 = 4 

■ Use the same number of standard deviations (SD 2 = SD,), calibrate safety 
factor for design 2 (SF 2 ) 

SF 2 = (60+ SD 1 *10.77)/60 

= (60+4*10.77)/60 = 1.718 

■ Calculate the required strength for design 2 based on the updated safety factor, 
SF 2 (original required strength = 60*1.4 = 84 ksi) 

Required Strength = 84 * 1.718/1.4 = 103.08 (> 95, not acceptable) 

Same Safety Factors May Not Be Appropriate for All Designs 
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EXAMPLE 


Assuming the same level of reliability, safety factor is a highly nonlinear function of 
load and strength uncertainties. This means that the safety factors must be calibrated for 
structures that are subjected to multiple loads to ensure adequate reliability level. 




Probabilistic technology 
can identify the critical 
areas that safety 
factors must be 
increased to meet 
required reliability and 
safety 

Probabilistic technology 
can reduce structural 
weight by identifying 
and reducing safety 
factors where possible 
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LATEST DEVELOPMENT IN PROBABILISTIC SOFTWARE TOOLS 


The following list includes the most recent available probabilistic software tools 
UNIPASS, ProFit, and ProModeler by Unipass Technologies 
NESSUS by NASA Glenn and SwRI 
ProFEA by ARA 
ANSYS PSD by ANSYS 
PROBAN by Veritas 
FEBREL by Boeing 



Latest Development in Probabilistic Software Tools 
Will Allow Companies To Maximize Their Business Success 


■ UNIPASS, ProFit, and ProModeler by Unipass Technologies 

■ NESSUS by NASA Glenn and SwRI 

■ ProFEA by ARA 

■ ANSYS PSD by ANSYS 

■ PROBAN by Veritas 

■ FEBREL by Boeing 

■ Etc 
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LATEST DEVELOPMENT IN PROBABILISTIC SOFTWARE TOOLS 


The general-purpose UNIfied Probabilistic Assessment Software System, 
UNIPASS , can be utilized independently, as a stand-alone software engine, and/or 
integrated with deterministic software tools to perform complex probabilistic analyses. In the 
analysis, UNIPASS™ provides the basis for modeling uncertainties, developing probabilistic 
process models, computing probabilities, identifying most likely outcomes, providing 
sensitivity data, identifying key drivers, analyzing risk, and performing sensitivity analysis, 
while the deterministic software tools may be integrated to provide the computational 
framework for constructing complex deterministic models. 

To ensure predictive accuracy, UNIPASS™ provides multiple algorithms, which the 
user can select. By comparing the results of several algorithms, a level of confidence in the 
predictions can be achieved. Furthermore, UNIPASS™, in addition to 1 1 gradient-based MPP 
identification methods, also provides a robust simulation-based search algorithm that 
identifies MPPs for discontinuous and/or non-differentiable limit-sate functions. 


UNIPASS Probabilistic Engine Provides Unmatched 
Capabilities for Performing Complex Probabilistic Analysis 

■ User friendly graphical user interface 

■ UNIX and Windows 95, 98, 2000, and NT operating systems 

■ 23 distribution types for modeling 4 classes of random variables including user- 
defined distribution 

■ 59 mathematical functions for modeling any complex event 

• Event model may be function of any variable or any previously defined function 

■ 3 analysis types including probability analysis, inverse probability analysis, and 
CDF/PDF analysis 

■ 4 different problem types including component, serial system, parallel system, and 
general system 

■ Numerous probabilistic methods in 6 categories for performing probabilistic analysis 
including FORM, SORM, SM, ISM, RSM, MVBM 

■ Generic and Customized Interfaces for easy integration with in-house and 
commercial codes 

■ Interface with MSC/NASTRAN finite element code 

Unipass Technologies 
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LATEST DEVELOPMENT IN PROBABILISTIC SOFTWARE TOOLS 


The probabilistic analysis begins by constructing process models and the probability 
distribution models. In addition to physic-based models, analysts often seek to construct 
models of processes, such as manufacturing processes, for which only some statistical data is 
available, and to construct These process models and probability distribution models can be 
constructed using our proprietary software tools ProModeler™ and ProFi™, respectively, 
utilizing available data. Using a general Bayesian framework, ProModeler ™ constructs the 
process models utilizing available data. This tool provides a framework for the analysis of 
uncertainties and model assessment by a Bayesian Updating Rule. Utilizing available data, 
ProFit™ identifies the best probability distribution model using a combination of the 
maximum likelihood method, three goodness-of-fit tests, and probability paper approach. 
The method of maximum likelihood involves taking as the estimate for each unknown 
parameter the value that appears most probable on the basis of the given data. The goodness- 
of-fit tests are objective techniques that provide a probabilistic framework in which it 
evaluates the adequacy of the distribution function. Probability paper approach is more a 
subjective method that determines whether or not the data contradict the assumed model 
based on a visual examination. This concept can provide a great deal of useful information in 
addition to an evaluation of the appropriateness of the chosen model. 



Provide Capabilities for Variable and Process Modeling 

■ ProFit engine identifies the best distribution type for a given data set 

• Identifies best distribution for a given data set comparing 22 distribution types 

• Performs 3 different goodness-of-fit tests 

• Provides probability paper 

• Estimates distribution parameters using method of moments and maximum 
likelihood method 

■ ProModeler Provide several techniques for building the process model and 
identifying patterns from given data using 

• Last-square method, regression approach, and Bayesian updating 


Unipass Technologies 


Figure 22 
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PROBABILISTIC SYSTEM DESIGN 


Probabilistic system design holds many opportunities to improve the total design 
process from concept to detailed design to service performance. Probabilistic design provides 
a rational basis for the linking of all of the interactive elements of system performance in a 
direct manner that accounts for the variability or uncertainties in all of the variables. 
However, there are major issues and challenges that yet must be overcome. The goal today is 
to provide an overview of the current state of probabilistic design while pointing the 
achievable work yet to be done. 


Probabilistic System Design: 
Issues and Challenges 


T. A. Cruse 
Private Consultant 
Pagosa Springs, CO 
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THREE MAIN MESSAGES 


The reality today is the probabilistic methods are already finding their way into the 
design process in various industries. The revolution in the design process is already 
underway. The presentation will also address the message that probabilistic methods are not 
yet ready to deploy to the design floor to support the certification of advanced, man-rated 
systems. Finally, the presentation will address ideas on the integration of probabilistic 
methods with some non-traditional methods (at least to those doing design). 


Main messages 

#> Probabilistic methods are being used today to 
design systems for industry 

^Probabilistic methods require further 
development to certify system designs 


Future design system developments require 
integration of probabilistics and non- 
traditional analysis methods 


May 30, 2001 


T. A. Cruse, Consultant 
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PRESENTATION OUTLINE 


Probabilistic methods are not just research. They have already been deployed to 
industry to support product design as well as process engineering, with great savings in cost. 
The joint AF/Navy JSF program has committed to probabilistic high cycle fatigue design 
requirements as the required technology to assure product performance and reliability. The 
presentation also includes reference to a study done by this consultant for the NASA Glenn 
Research Center as part of the now-defunct NASA ISE program. 

However, the basic tools and methods used in probabilistic design still require 
significant improvements in order to adequately support the design process for advanced 
aerospace systems. The key issues and challenges that remain included robustness and error 
bounds for all algorithms, the ability to predict confidence (or assurance) bounds on the 
predicted system outcome, and model verification and validation methods. 


Presentation outline... 

4 Where we are today 

* Some industrial successes 

■ Air Force probabilistic-HCF program 

■ Non-deterministic, non-traditional methods 
(NDNTM) for design study (NASA/GRC) 

# Issues and Challenges 


■ Robustness and error bounds for models 
h Confidence/Assurance bounds for data/models 
* Verification and Validation procedures 
^Conclusions and recommendations 

May 30, 2001 T. A. Cruse, Consultant 
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PREDICTIVE RELIABILITY ENGINEERING 


Working with the Los Alamos National Laboratory, Delphi Automotive has deployed 
the PREDICT system that uses a probabilistic network approach to support the performance- 
based, reliability growth design process from concept to field deployment. Probabilistic 
modeling of the manufacturing processes at Procter and Gamble, also using LANL 
developed technologies, identifies processing improvements that achieve bottom-line 
benefits. That effort has been so successful that P&G is now marketing the technology to 
others. References are given to some of these keys, recent applications of technologies 
developed at Los Alamos National Laboratory. The technologies were developed to support 
the nation’s weapons reliability requirements where analysis must substitute for full-scale 
testing. 

Anon., “Proctor & Gamble starts peddling a “top secret” manufacturing technology,” 
Manufacturing News, Vol. 8, No. 6, pp. 1, 6-8, Friday, March 30, 2001. 

Kerscher, W. J., Booker, J. M., Bement, T. R., and Meyer, M. A., “Characterizing 
reliability in a product/process design assurance program,” Los Alamos National Lab Report 
LA-UR-97-4072, Proceedings of the International Symposium on Product Quality & 
Integrity, Jan. 19-22, 1998, Anaheim, CA. 


Predictive reliability engineering 


#Los Alamos National Lab has focused on 
deploying reliability based design tools to 
industry 

■ Proctor & Gamble project simulates reliability of 
manufacturing process to within 1% [Mfg News, 
March 30, 2001] 

■ Delphi Automotive uses PREDICT methodology for 
total mechanical system design [LA-UR-97-4072] 

^Critical contributions focus on the integration 
of disparate types of data and information 


May 30, 2001 
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LANL PREDICT METHODOLOGY OVERVIEW 


The PREDICT system has as its operating core the prediction of system performance 
(e.g., reliability) from conceptual design through field deployment. While the application to 
Delphi Automotive focused on reliability growth during the design and deployment process, 
aerospace design typically starts with a “successful” design on paper that meets the reliability 
goal but at a level of excessive weight, or other performance shortfall. The point is that 
predictions are made and tracked throughout the design process. Further, the range of 
uncertainty on the performance metric is identified and tracked along with its principal 
drivers. The goal is to increase reliability while decreasing the uncertainty range on that 
reliability. The PREDICT system tracks the metrics along with the drivers so that 
development and test investments are made where they can have the greatest impact on 
improved design. 

As a normal part of the design process, the reliability may be adversely changed as 
the result of new information or data. That new information is fed into the Bayesian network 
to provide rigorous updates to the system performance. The critical technology to me in the 
LANL effort is the ability to integrate disparate forms of design and experimental 
information from expert opinion to test data. 


LANL PREDICT methodology 


■! Tracking Reliability Through System Lifetime 



Statistical Sciences Group 

May 30, 2001 
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AF PROBABILISTIC HCF PROGRAM 


The JSF turbine engine designs use integrally bladed rotors whose lightweight and 
flexibility means that dynamic modes are fully coupled across and around the rotor. Such 
dynamic coupling can result in “tuned absorber” dynamic response such that all energy feeds 
into a small region resulting in rapid structural failure. Deterministic design methods have 
been found to be incapable of controlling this phenomena and the commitment has been 
made by industry to step up to a full probabilistic design that links to manufacturing 
variability. 

HCF design has been a major AF field failure problem for over two decades. Rotor 
LCF has largely been controlled through the damage tolerance design approach but HCF 
technology has lagged. The controlling design requirements for all engine structures is 
defined by the Engine Structural Integrity Program (ENSEP) guidelines contained in Mil-Std- 
1783A. That standard is now being updated to include probabilistics. The zero th level change 
that has been accepted by industry is a probabilistic resonant frequency avoidance criteria. 

The AF has joined with NASA to continue what had started as a NASA ISE program 
to infuse some of the LANL PREDICT technology into the AF HCF program. In particular, 
the effort is demonstrating how to elicit design and operating information from industry on 
the sources of and nature of the variability in the aeromechanics drivers and structural 
response for bladed rotor HCF failures. 


I Sti WSSSk 


AF is Committed to probabilistic 
basis for HCF life prediction 


# Requirement driven by integrally bladed 
rotors that cannot meet reliability 
requirements otherwise 

#HCF identified as the major structural 
reliability problem in the field 

^Preliminary, zero th level probabilistic design 
requirements are being adopted 

Joint NASA/AF/Industry effort to apply 
portions of the LANL/PREDICT system to HCF 
problem 


May 30, 2001 
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NASA/GRC ISE-FUNDED STUDY OF NDNTM FOR SYSTEMS DESIGN 


This funded study was performed in CY 2000 and a final report is available through 
the Program Manager, Dr. C. C. Chamis. The context for the study undertaken by this 
consultant is the AF HCF program, the Consultant’s experience and work in probabilistic 
design, a comprehensive literature review, and a personal belief that NDNTMs are required 
for the next generation space access vehicle system design. The study included a review of 
NASA’s design technology base, current design environment studies in industry, and 
technology capabilities and efforts at a number of key small business or university sites. 

The conclusion of the study is that we have no real current need for new probabilistic 
methods but that we have many important tasks to complete in order to deploy non- 
deterministic design into aerospace vehicle design practice. Another key conclusion reached 
by this consultant is that the nation lacks the critical leadership necessary to achieve a truly 
inter-operable, information based design environment such as envisioned by the NASA ISE 
effort. At this time it appears that every major aerospace firm is working on its own vision 
for such systems and these are not likely to work together. 


. . 


NASA/GRC study on Non-deterministic, 
Non-traditional Methods 



Figure 7 
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NDNTM STUDY PUBLISHED KEY RECOMMENDATIONS 


The conclusion of my NDNTM study for NASA was that the future space access 
systems designs must integrate design information in innovative ways that include the 
variability and uncertainty issues upon which probabilistic design methods focus. The design 
environment is fundamentally driven by the need to integrate disparate forms of information 
from data to judgment. The design environment also requires the ability to work with many 
interoperable tools that might be generated by diverse sources and integrated at the desktop. 
The study proposed an integrated non-determini Stic design environment. 


Principal recommendations 

^Future, 3 rd Gen system design will 
integrate variability and uncertainty 

■ Design depends on integrating disparate 
forms of information and analysis 

■ Approach requires a standard software 
engineering environment with 
interoperable tools, not packaged software 

^Integrated non-deterministic design 
environment was recommended 


May 30, 2001 
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INTEGRATED NDNTM DESIGN ENVIRONMENT CONCEPT 


The focus in this graphic is on non-deterministic design using non-traditional 
methods. The chart shows in blue ovals the technology areas for which details are proposed 
in the NDNTM Final Report. Elements include intelligent interfaces, Bayesian networks, 
fuzzy logic for data/information fusion, and other non-traditional technologies. However, the 
central processing element of the NDNTM is mathematically rigorous probabilistic methods. 
Such algorithms as fuzzy logic were not found to be suitable or appropriate for the core 
analysis algorithms. 


Integrated, non-deterministic 
design environment needed 



Figure 9 
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SIGNIFICANT CHALLENGES REMAIN 


The current technologies used in probabilistic design appear to meet the design 
environment requirements in terms of basic capabilities. However, these methods lack the 
necessary robustness to be deployed to the design floor. Fast probability methods fail to 
converge for certain problem formulations, the output predictions have unstated or unknown 
accuracy, and the methods still require that the users be probabilistic experts. 

The future design environment for advanced aerospace systems will involve many 
types of data and information to be integrated into the prediction system. The effective 
approach is to more fully utilize expert systems and their knowledge bases to provide 
interfaces to the tools, to assist in data preparation, and to provide “error traps” for complex 
modeling. Reliability networks are needed to represent the overall system being designed, to 
provide “what-if” modeling capability, to provide weighting functions for decisions on 
development investments, and to rigorously support reliability updating as new information 
is obtained. A critical need is to have the ability to merge soft data (engineering judgment) 
along with crisp data (e.g., experimental data) in rational ways that minimizes bias while 
fully retaining evaluative links to the data sources. 


V,, 'T; 


Significant challenges remain 


Probabilistic tools are not yet robust 
■ Methods sometimes fail to converge 
a Modeling accuracy levels are not known 
a Applications require expert users 

^Integration with non-traditional methods is 
required 

a Reliability networks with updating 
a Integration of soft and crisp data 
a Expert systems and knowledge bases 


May 30, 2001 
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FAST PROBABILITY ALGORITHM ROBUSTNESS TESTED 

As part of the AF HCF program, the Consultant applied some of the standard fast 
probability integrations, as provided by Southwest Research Institute, to a simple ID 
oscillator design problem posed by GE Aircraft Engines (GEAE). The ID model was sharply 
peaked in the form of the shown limit state. A mathematically equivalent form of the same 
limit state can be derived that is essentially an inverse of the shown state. That equivalent 
form did not have the indicated problems in computing predicted reliabilities. 

The GEAE problem had seven random design variables. They derived eight design 
cases that had differing nominal conditions for each of the seven variables. Using a design of 
experiments (DOE) approach, GEAE determined values of the design variables for each 
design case that had low probabilities of failure (third column). The Monte Carlo method was 
used with different numbers of simulations (second column) to make what were taken to be 
converged reliability predictions. The remaining three columns show the Advanced Mean 
Value Plus (with iterations), First Order Reliability Method, and Second Order Reliability 
Method predictions. In the form of the peaked response model of the ID system, each of the 
methods failed for one or more of the DOE cases. The problems are not associated with the 
so-called multiple-limit state problem for fast methods. Rather, they seem to be purely 
numerical problems associated with a poorly resolved limit state. 



•Strongly-peaked response function form for limit state 
•The limit state form resulted in poorer numerical behavior 
•Problem is similar to what occurs in numerical optimization 
algorithms 


DOE Case 

N_sims 

MC (TAC) 

AMV+ 

FORM 

SORM 

2 

600K 

0.00017s 

0.00016 

0.0000 

-10^26 

4 

5200K 

0.00763 

0.00764 

~10 A -32 

-10M2 

6 

600K 

0.00059 

0.00059 

0.0000 

0.0000 

8 

200K 

0.01440 

0.99980 

-10^-25 

0.0000 

10 

600K 

0.00369 

0.00356 

0.00309 

0.0037 

12 

200K 

0.03040 

0.02940 

0.99990 

0.9999 

14 

200K 

0.02448 

0.02335 

0.02200 

0.0239 

16 

200K 

0.08477 

0.08210 

0.07940 

0.0834 
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SMART ENVIRONMENTS NEEDED 


Some of the conclusions reached by the Consultant regarding the current probability 
integration algorithms are indicated. The fact is that fast methods are really faster than Monte 
Carlo. There is little doubt to this observer that fast methods will be needed for future 
aerospace systems design. However, their deployment will not be accepted until the 
shortcomings are rigorously addressed. Research sponsors and program managers must link 
the development of future tools to requirements that address these needs. 

Monte Carlo methods will also be required. While typically taken to be “truth” there 
are significant capabilities that these methods must also contain in order to be used in this 
future design environment. The needed capabilities include the ability to capture the so- 
called probabilistic sensitivity factors that are critical to the design process. Such factors are 
used to narrow the problem space, to allocate information resources, and to compute 
confidence intervals. Monte Carlo methods must also have their own automated simulation 
error controls that adaptively adjust the simulation numbers to the outcome probability of 
failure results. 

All methods used must make modeling error bound estimates for each problem. The 
error bounds are required to support system certification and to compute confidence interval 
estimates. 


All probability integration algorithms 
require smart operating environments 

♦"Fast" probability algorithms are faster, but... 

* Robust convergence is required 

■ Algorithm error bound estimates needed 

* "Erroneous use" traps needed 

# Monte Carlo algorithms working with 
response surface models are much more 
robust, but... 


> Sensitivity factors needed near failure condition 
(MRP) 

* Automated "error" convergence control required 



All methods require known modeling errors 
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CONFIDENCE INTERVALS ARE REQUIRED 


Classical statistical confidence intervals are mathematically linked to the quantity of 
experimental data. As such, statistical confidence intervals are not appropriate to the future 
NDNTM design environment. The fact is that aerospace vehicle design, as opposed to 
electronic device manufacturing, cannot be evaluated by replicated system failure testing! 
Yet, confidence intervals are still required - we need to be able to answer the question of 
“how good is our estimate?” 

Bayesian (belief) networks are the effective mathematical means for computing 
engineering confidence intervals (as opposed to statistical confidence intervals). Some would 
call these intervals assurance intervals to make the distinction clearer but the word 
“confidence” appears to be the most meaningful to engineers. Bayesian networks allow many 
kinds of answers to the indicated questions to be integrated in estimating the range of 
outcomes. 

Certainly, the key issues addressed in the Verification and Validation process for 
analysis methods also contribute to making such confidence or assurance estimates. 


flilSI 

The next big issue is calculating 
"confidence" or "assurance" 


♦ Statisticians have standard tools for dealing with 
finite data, e.g., confidence intervals 

■ Statistical confidence intervals typically drive design data 
requirements 

* Statistical confidence intervals define data quality 

0 Design must analytically account for the "quality" of 
data (information) and models 

■ How "confident" are you about your data (information)? 

* How "confident" are you about your data model? 

■ How "confident" are you about the physical model? 

0 Assurance is a by-product of a process approach to 
"verification" and "validation" 


May 30, 2001 
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DISTRIBUTIONS HAVE DISTRIBUTIONS 


As a simple example, I illustrate the effect of uncertainty on the seven input 
distributions used for the GEAE ID oscillator test problem. Again, I acknowledge the 
support of SwRI through the use of their Nessus code to make these calculations. The 
algorithm is base on outer “do loops” of Monte Carlo simulations that account for the 
variability in the input variables. 

In each case shown, each random design variable’s distribution parameters, the mean 
and the standard deviation, was assumed to have a uniform distribution. The “width” of the 
uncertainty, or its interval, was defined by the coefficient of variation of that parameter. Two 
cases were taken where the mean and standard deviation had intervals of 5%, 10% 
respectively, and 0.5%, 1.0% respectively. The results are stated in terms of the two-sided 
(upper and lower bounds) for the predicted probability of failure of 0.0144. That is, how 
large (or small) might the probability of failure be if we don’t have perfect information on the 
input variables? The simulations do not include the effect of modeling errors! 






Distributions on distributions 


Input data models are 
themselves uncertain 

GEAE HCF Problem 
- MC (200K): P f = 0.0144 
Uncertainties assumed 
on distributional 
parameters 

* Uniform for p, a 

* COV for p = 5% [0.5%] 
■ COV for a = 10% [1%] 


I Nessus code courtesy of SwRI j 

MC simulation of 
simulation results: 

* Nessus/Confidence 

♦ 90% LB: P f = 0.0000044 

* 90% UB: P f = 0.425 

♦ [90% LB: P f = 0.0081] 

* [90% UB: P f = 0.0238] 

Still must account for 
modeling errors 
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VERIFICATION AND VALIDATION 


Design analysis methods themselves have uncertainties. Verification addresses 
whether or not the computational methods that are used accurately represent the assumed 
model. Validation addresses whether or not the model accurately represents the underlying 
physics that is being modeled. Both steps are critical. 

System certification addresses both the process and the outcome. An example is the 
FAA certification of the design of critical rotating parts in commercial aircraft turbine 
engines. The certification process requires a complete verification and validation of the tools 
used - deterministic tools today - to certify the safe life predictions for these critical engine 
components. 

I am certain that such verification and validation requirements will exist for the 
NASA space access program. Such methods are now being developed in detail for the 
national nuclear weapons certification program. The AIAA has published a V&V guide for 
CFD while the ASME has just approved a code committee action for finite element 
programs. No such effort currently exists for probabilistic methods. 


Verification and Validation methods 
are required for new system design 


#AII system behaviors have uncertainties 
« Verification addresses coding of models 
■ Validation addresses accuracy of models 

# Certification is based both on the process 
and the outcome 

# System level V&V will be required for 3 rd 
Generation RLV certification 

#Sandia/DOE, AIAA (CFD), ASME (FEM) 
V&V efforts ongoing and can be used 


May 30, 2001 
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CONCLUSIONS 


Probabilistic design is not some future thing. It is here today. It is the right approach 
to some of the most critical design issues of risk-based decision making, cost reduction, and 
testing. However, the design environment needs real work to be done in order to meet the 
needs of certifying complex aerospace systems. Finally, system certification must include 
work on the verification and validation of the probabilistic methods that we will use. NASA 
Program Managers are in a position to address many of these needs in their future research 
procurements. 




Conclusions and recommendations 


41 Probabilistic design is here today 

■ Rational process to support risk based requirements 

■ Rational process to evaluate cost reduction vs. risk trades 

■ Rational process to define testing requirements 

#> Probabilistic design environment needs work 

■ Integration of disparate information & models needed 

■ Robustness and accuracy issues must be resolved 

■ Smart design environment needed 

4 Verification and Validation (V&V) effort needed 

■ Quantify the role of modeling uncertainties 

* Quantify the effect of uncertainties on predicted 
performance 

May 30, 2001 T, A. Cruse, Consultant 
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http: / /www. asdl . gatech, edu/ 



Motivation 


. :■ V v : ; .. 


Moving from deterministic design to robust/probabilistic design methods 
amounts to an admission that uncertainty exists and has a significant impact 
on system performance 


Probabilistic Design 


Want to analytically answer the questions: _ 

- How much design margin is really necessary? 

- How do design parameters impact the uncertainty in performance? 

- What can be done to reduce this impact? c=£i32= 


- wnai can ue uuuc 10 leuuce ims unpdu: i | Robust Design | 

Obstacles to implementation: 

- Organizational inertia 

- Lack of probabilistic analysis tool to bridge the gap between deterministic and 
probabilistic methods 

- Computational Costs, if not approached intelligently 


br. Dimitri Mavris 
Georgia Institute of Technology 
Atlanta, GA 30332-0150 
'.asdl.qatech.edu 


J^SD 



143 




The Design Process Paradigm Shift 


A paradigm shift is underway that attempts 
to change the way complex systems are being 
designed 

Emphasis has shifted from design for 
performance at any cost to design for 
affordability 

There is a need for a multi-disciplinary 
approach to the problem based on more 
sophisticated higher fidelity tools 

Forecasting with a high probability of 
success the economic viability of the system 
in the early phases of design appears to be the 
key to success 

Due to the life cycle implications of this 
approach a need exists to create an 
environment that virtually designs, tests, 
certifies, manufactures, and operates the 
system, while accounting for design 
ambiguity, uncertainty and risk 


,M' ‘A 



mmm Today's Design Process 
"—Future Design Process 
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Analysis Prototype Redesign Product 

and Detail Development Release 
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Affordability- Making the Right Decisions Early 


Pre-milestone 0 

Determination of 
Mission Need and 
Deficiencies 


Concept 

Exploration 


Acquisition Timel i ne 

[p Phase I | Phase II Tl j j " Phase III ' ~ 

Program Definition 8 Engineering & B | Production, 

and Risk | Manufacturing 1 1 Deployment, and 

Reduction H Development B B Operation Support 


Design Freedom 

i 


Knowledge 
becomes available 
when time to make 
decision 


w Cost Committed 


Today 

Future 


Knowledge 


Requirements 

Definition 


Conceptual 

Design 


il Preliminary 

Design 

Design Timeline 


Detail Design + Manufacturing 
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Response Surface Methodology (RSM) 

RSM is a multivariate regression technique developed to model the 
response of a complex system using a simplified equation 
Regression data is obtained intelligently through the Design of 
Experiments (DoE) techniques 

RSM is based on the design of experiments methodology which gives 
the maximum power for a given amount of experimental effort 
Typically, the response is modeled using a second-order quadratic 
equation of the form: v 


k k - 1 k 

ST z. . .2 , V 1 


R = K + X h i x i + X b u x i + X X Vi- 

i= 1 /=! i=l 7=i +l 


Where, 



b ± are regression coefficients for the first degree terms 
b i± are coefficients for the pure quadratic tenns 
by are the coefficients for the cross-product terms 
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Design of Experiments 




Purpose: Minimize number of experiments required for desired level of resolution ! 


Design of 
Experiments 

For 7 
Variables 

For 12 
Variables 

Equation 

Full Factorial 

2,187 

531,441 

3 d 

Central 

Composite 

143 

4,121 

2°+2n+l 

Box-Behnken 

62 

2,187 

- 

D-Optimal 

Design 

36 

91 

(n+l)(n+2)/2 
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Parametric Description of a Wing Planform 


Other Design Variables 


xwmg 
t/c at root 
t/c at tip 
Nacelle Scaling 
Horizontal Tail Area 
CL Design 

Root Airfoil (loc. max., thickn.) 
Tip Airfoil (loc. max. thickn.) 
Nacelle X-location 
Wing Reference Area 


Planform Variables , 
(Normalized by Span)/ 


< X4 ' Y1 > X5.0 
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Possible Wing Planforms 
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Options for Probabilistic Design 



Fast Probability Integration (FPI) 

X n 

y j y 2 

• FPI manages program execution while 

X. \ 

handling up to 100 deterministic fo) or 

X 5 / 

■^fpTx/ 

/ piODclDIlISlIG vYj/ VctMclDISS, Willi OdpaUllIly TUi 

expansion 


NEPP \ 

t • Establishes design feasibility 

X, 


I • Identification of most critical constraints 


FLOPS 

1 • Creates probabilistic sensitivity derivatives 
S and CDFs for each objective & constraint 

X 3^\ 

ALCCA J 

■ • Assessment of new technologies impact 


deterministically or probabilistically 

X 2 / 


• Probabilistic solutions for a set of design 

X 1 


variables subject to uncertainty 

• Identification of feasible 

and/or robust solutions, by assigning random 

distributions to each design variable, within the range of applicability, and 
allowing for operational and manufacturing uncertainty 
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Technical Feasibility vs. Economic Viability 


Feasible and 
Viable Solution 


Non-Feasible, 
Non-Viable Solution 


Desired 

Solution 


Objective or Constraint 


Target 


Baseline 

Mean 


Objective 

Constraint 


yhh. 



Addressing Technology Benefits, 
Penalties and Confidence 


IMMMMi 


1 . Create functional 
relationships between 
Objectives/Constraints 
and technology metrics 

2. Model technology 
benefits and penalties 
through metric “k-factors” 

3. Assign probability 
distributions to the 
technology metrics to 
develop confidence 
estimates and CDFs 


Objective = f(L/D CI1 
Con s tr a i n t = f(L/D CI 

i 


(Step Change 
in Mean) 


(Variability) 


MMH/FH, . . .) 
MMH/FH, . . . 

t 







New Constraint CDFs through Technology Infusion 




Technologies have the affect of shifting the response distribution such 
that an acceptable confidence in meeting the constraint or objective is 

obtained 


Examine Feasible Space 


Based 

M&S 

Monte Carlo 


Constraint 

Cumulative Distribution 
Functions (CDFs) 






Asd 



JPDM- Mapping the Solutions 


Joint Probability Decision 
Making (.JPDM) is an 
enabling technology that 
allows for the probabilistic 
assessment of multi-criteria 
problems 


Three alternative 
solution concepts, with 
probability bands due to 
growth and technology 
uncertainty 





Criterion 1 
or Requirement 1 
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Technology Identification Evaluation Selection 



'MADM- 


-/(CV-NV^) 

Andym + Metwnodd ■ 
* Monte Carlo 


Morphological Matrix 


Technology 

; Frontier!) • 


Customer 

Requirement 

Tarots 


Level of Confidence; 
,lem RSI- or direct link 
(discipline specific) 


Customer 
Requirements, 
Budget, 
Schedule (E1S) 


Maries 


Tech lit Metrics i 


“Best” Fnrrtities 
of Technology 
Alter iicUives 


Compatibility Matrix 


Technology Readiness, or 
maturity, Level (TRL) 
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Morphological Matrix 
(Alternative and Design Spaces) 
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EXAMPLE APPLICATION 


The TIES method developed a few years ago in response to some very aggressive 
vehicle concepts that could not meet future performance or economic objects, especially with 
present day technologies or geometric perturbations. New technology infusion was the only 
option. Yet, the cost conscious industry was very concerned with investment cost and risk 
associated with developing and infusing new technologies that have a great deal of 
uncertainty. Thus, a means to quantify the impact in terms of performance, cost, and risk in 
the early phases of design was needed. 

Hence, the Technology Identification, Evaluation, and Selection method was created. 


Example Application 

• Aggressive economic and performance objectives of future 
concepts likely cannot be meet with present day technologies 

• A “focus on the bottom line” has forced many aerospace 
companies to dismiss new, innovative, and revolutionary 
designs due to the potential risk of profitability loss 

• Yet, if a technology can be shown to improve a system at low 
risk, it may buy it’s way onto the aircraft 

• A comprehensive and structured process, applicable to any 
system, was needed to quantify and forecast the impact of 
emerging technologies while accounting for technological 
uncertainty. This method is 

TIES 


Technology Identification, Evaluation, and Selection 
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Problem Definition 

Societal Need: 

Design Rectuirements: 

Desire for a next generation supersonic aircraft 

Through QFD and brainstorming 

Increased commercial traffic growth 

exercises, the customer requirements 

Increased comfort, safety, and affordability 

were mapped to quantifiable 

Potential Concept Class: 

engineering parameters 

High Speed Civil Transport 



(Concorde class derivative ) 





Target/ 




Corst-rainr 

L nits 


Perform!) not-. 




Approach S peed (V ) 

< 155 

kts 


FAR 36 Stage III Flyover Noise (FON ) 

<106 

EPNLdB 

Criteria 

landing Field Length (Landing FL-) 

<11.000 

11 

FAR 36 Stage 1H Sideline Noise (SLN) 

<103 

EPNI/1B 


Takeoff Field Length (TOFL) 

<11. (XX) 

ft 


Takeoff Gross Weight (TOG W;. 

< 1,000,000 

lbs 


Economics 




Acquisition Price (Acq $) 

minimize 

FY96 SM 


Research, Development, Testing, and 
Evaluation Costs (RDT&E) 

minimize 

FY96 SM 

Economic 

Average Required Yield per Revenue 

<$0.10 

FY96 5 

Criteria 

Passenger Mile ($/RPM) 

Total Airplane Related Operating Costs 


FY96 $ 
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minimize 
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DEFINE CONCEPT SPACE 


With the requirements defined, a potential class of vehicles must be defined. A 
structured means of doing so is with a Morphological Matrix. The morphological matrix is 
nothing more than a decomposition of all possible contributing elements of the system. It is a 
means to brainstorm and think out of the box for potential solutions to the problem. 

For example, the project manager could bring together all of his experts and 
decompose the system. Do we want a wing and tail vehicle? Or a wing and canard? And so 
on. If you do this for each element of the system, then you have effectively defined the 
alternative concept space which may have mission parameters, technologies, and so on. 

Once this matrix is sufficiently defined, one must establish a baseline to continue on 
with the TIES method. You do this by selecting one element from each row like the circled 
items, usually present day capabilities. This is your baseline that you will do all deviations 
on. 

Next, that system is further decomposed into geometric and propulsive parameters 
that will define the design space to be investigated for feasibility. 


1HS! 


Define Concept Space: 

Alternatives Space 


Space of Alternatives 

- Began with a Concorde-class aircraft and modified based on market studies 

- Decomposed vehicle class into system characteristics and characteristic 
alternatives to define the component alternatives and potential technologies 
through a Morphological Matrix 

- Baseline configuration (circled) established from the characteristic alternatives 
representative of present day capabilities 

- Mission defined based on market requirements 


Vehicle Class 
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2 

2.2 
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2.7 

Propulsion 
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Mid Tandem 

Hade 

Materials 





Combustor 


ROL 

LPP 


Nozzle 
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Acoustic Liner 

o 

<U 

< 
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■amii 

CC 
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Active Control 
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1 
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DEFINE CONCEPT SPACE: DESIGN SPACE 


With the requirements defined, a potential class of vehicles must be defined. A 
structured means of doing so is with a Morphological Matrix. The morphological matrix is 
nothing more than a decomposition of all possible contributing elements of the system. It is a 
means to brainstorm and think out of the box for potential solutions to the problem. 

For example, the project manager could bring together all of his experts and 
decompose the system. Do we want a wing and tail vehicle? Or a wing and canard? And so 
on. If you do this for each element of the system, then you have effectively defined the 
alternative concept space which may have mission parameters, technologies, and so on. 

Once this matrix is sufficiently defined, one must establish a baseline to continue on 
with the TEES method. You do this by selecting one element from each row like the circled 
items, usually present day capabilities. This is your baseline that you will do all deviations 
on. 

Next, that system is further decomposed into geometric and propulsive parameters 
that will define the design space to be investigated for feasibility. 


Design Space 

- Design space was defined with geometric and 
propulsive variables from a decomposition of baseline 

- Ranges established to capture as many configurations as 
possible, from an arrow to a double-delta and push the 
state-of-the-art in propulsion capabilities 

- Initial feasibility is sought within this space 


Define Concept Space: 

Design Space 


Variable 

Minimum 

Maximum 

Units 

Description 

SW 

7500 

9000 

ft 2 

Wing area 

TWR 

0.29 

0.33 

~ 

Thrust-to-weight ratio 

TIT 

3000 

3400 

°R 

Turbine Inlet Temperature 

FPR 

3.5 

4.5 

~ 

Fan Pressure Ratio 

OPR 

18 

21 

~ 

Overall Pressure Ratio 

CLdes 

0.08 

0.12 

~ 

Design lift coefficient 

X2 

1.54 

1.69 

~ 

LE kink x-location* 

X3 

2.1 

2.36 

~ 

LE tip x-location* 

X4 

2.4 

2.58 

~ 

TE tip x-location* 

X5 

2.19 

2.37 

~ 

TE kink x-location* 

X6 

2.18 

2.5 

~ 

TE root x-location* 

Y2 

0.44 

0.58 

~ 

LE kink y-location* 

t/c_root 

3 

5 

% 

Wing root t/c ratio 

t/cjip 

2 

4 

% 

Wing tip t/c ratio 

SHref 

400 

700 

ft 2 

Horizontal T ail area 

SVref 

350 

550 

ft 2 

Vertical Tail area 



Example configurations 


M -< ◄ 


* Variables Nondimensionalized by wing semi-s 
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MODELING AND SIMULATION: VEHICLE MODELING 


Once the definition of your design space is established, you need an environment that 
will model the vehicle and allow you to see how the customer requirements are influenced by 
the design space. Typically, one uses a sizing and synthesis tool. Yet, standard tools must be 
enhanced for non-conventional configurations like an HSCT. This is due to the fact that most 
tools are based on historical data and must be enhanced with higher fidelity analysis tools to 
give reasonable results. In this example, the aerodynamics internal to the sizing tool were 
replaced with higher fidelity aerodynamic metamodels which would capture the entire design 
space under consideration. 

This then creates an HSCT specific, physics-based modeling and simulation tool and 
you are ready to investigate your design space. 


Modeling and Simulation: 

Vehicle Modeling 


M&S environment: 

• Relates responses to inputs via a physics-based 
M&S environment 

• Metamodels are employed to facilitate the use of 
higher-fidelity analysis for unconventional 
configurations 


Response =/ (design variables), or 

=/ (technology “k” factors) 
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ALCCA (Aircraft Life-Cycle Cost Analysis): Developed by 
NASA -Ames and enhanced by ASDL; calculates life-cycle costs 
and airline economics for transport aircraft. 

FLOPS (Flight Qptimiz.ation System): A NASA-Langley 
vehicle synthesis and sizing code, well-suited for the conceptual 
and preliminary design of subsonic transport aircraft. 
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INVESTIGATE DESIGN SPACE: SYSTEM METRIC SENSITIVITIES VIA 

PREDICTION PROFILES 

The RSEs can be visualized within the JMP statistical package with the prediction 
profile feature shown here. As a side note, system metrics are synonymous with customer 
requirements. 

I want to point out the key info here in the prediction profile. First of all, the metrics 
are listed on the left and the design variables on the bottom in a non-dimensional form where 
-1 is the minimum and +1 is the maximum. The redline, or hairline, corresponds to the 
current value of the design variable settings. And the metric values that result from the 
current design variables is shown in green. As you move the hairline or change the design 
variable value, the metrics automatically update. 

You can see the influence of each variable on the metrics by the magnitude and 
direction of the slope. For example, looking at thickness to chord at the tip we can see that as 
we increase the thickness, the TOGW also increases. 

You can also optimize your design variable settings with the desirability feature. You 
can place the constraint values or the direction that you want a metric and then determine the 
optimal settings of the design variables. I have shown here the design variable non- 
dimensional values that correspond to a maximum desirability. 

You can also immediately identify the upper and lower bounds of the design space, 
but you are not sure how close the space is to either value. Is most of the space near the lower 
bound or the upper bound? To do so, one simply extract the RSEs that are behind the 
prediction profile and execute a Monte Carlo simulation on those equation with the design 
variables allowed to vary anywhere within the defined ranges. 


Investigate Design Space: 

System Metric Sensitivities via Prediction Profiles 

Metric Upper/Lower bounds of Influence of parameter on response 


Responses 

1 srg 

/ the design space Hairlines move and update (either T or i or no influence) 

/ ^ responses in real-time Tlie larger the slope, the greater the influence 
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HSCT FEASIBILITY ASSESSMENT: DESIGN SPACE REPRESENTATION 


A result of examining the design space is a cumulative distribution representation of 
you metrics as a function of design variables. Let me explain the information that you get 
from doing this. 

First, you can see the bounds of the design space and how much and or how close the 
space is to your metric constraints. 

Second, you can readily identify the technical feasibility of the design space by 
looking to see how much of the space is on the feasible side of the constraint target and then 
read off the Probability value. For example, 4.6% of the space can satisfy the TOFL 
requirement. 

Third, if you look at the metric value at 0 probability, that value is the best that you 
can EVER achieve with the design space you’ve defined. 

And finally, and most important, you can see which constraints are killing you and 
are the “show-stoppers” for the program. In this case, none of the space can satisfy the 
sideline noise AND the space is very far away from the target. Hence, some significant 
improvements are going to be needed to obtain a feasible solution. 

So, what can you do? 
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TECHNOLOGY IDENTIFICATION 


Based on the fact that technologies are needed to improve the system feasibility, 1 1 
applicable technologies and their associated TRLs were identified either through a literature 
search or provided from various entities. As you can see, most of the technologies considered 
are at a TRL of 3. And if you recall from the previous slide, that implies that there is a lot of 
uncertainty and the anticipated impact has a low chance of being achieved 



Technology Identification 

• 1 1 technologies were established from the needed improvements identified in the 
system feasibility investigation, in addition to enabling technologies 

• Associated Technology Readiness Levels (TRLs) were established from a 
comparison of the current research activities to the TRL descriptions 

• Technology Compatibility rules were determined from brainstorming sessions 

(Name) Technology 

TRL Purpose 

(Tl) Composite Wing 

3 Wing weight reduction 

(T2) Composite Fuselage 

3 Fuselage weight reduction 

(T3) Circulation Control 

4 Increased low speed performance 

(T4) Hybrid Laminar Flow Control 

3 Cruise drag reduction 

i= 32®>-(T5) Environmental Engines 

3 CReduce noisgXuel burn, and emissions 

(T6) Advanced Flight Deck Systems 

4 Synthetic vision removes fuselage nose droop 


weight penalty 

(T7) Advanced Propulsion Materials 

3 High temp, materials, reduced engine weight, lower 


fuel burn 

(T8) Integrally Stiffened Aluminum Wing 

4 Wing weight and part complexity reduction 

Structure 


(T9) Smart Wing Structures 

3 Reduced flutter and wing weight 

(T10) Active Flow Control 

3 Cruise drag reduction 

"“““►(Til) Acoustic Control 

3 C2^fse'snppresk»5> 
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TECHNOLOGY IDENTIFICATION: TECHNOLOGY PRESENTATION 
THROUGH MAPPING TO “K” FACTORS 


Unfortunately, advanced technologies are difficult to assess in an integrated design 
environment. As mentioned earlier, synthesis/sizing tools are typically based on regressed 
historical data, which limits or removes the applicability to revolutionary concepts or 
technologies. However, the impact of generic technologies can be quantitatively assessed 
with technology impact factors, denoted as "k" factors herein, in the early phases of design. 
These "k" factors modify disciplinary technical metrics, such as specific fuel consumption or 
cruise drag that result from a sizing tool. The modification is essentially an incremental 
change in the technical metric, either enhancement or degradation. In effect, the "k" factors 
simulate the discontinuity in benefits and/or penalties associated with new technologies. 
This assessment is performed in the Technology Impact Forecasting environment. 



Technology Identification: 

Technology Representation through Mapping to “k” Factors 

a ' m mm® m mmm a mmmmmmmmmmmmmmmmmmmm 

• Formulation in terms of elementary variables does not lend itself 
to disciplinary or multidisciplinary technology assessment 



Conventional 
Aero. Analysis 


New Tech. 
Aero. Analysis 


Conventional 
models break down 

? %A L/D 


• The assessment of new technologies must be addressed through 
the disciplinary metrics (or technology “k” factors) since a 
mathematical formulation is not yet available 


constraints/objectives - /(k_L/D sub , k_L/D sup ,k_ C Lmax , k_Tl, k_SFC sub , ...) 
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TECHNOLOGY IDENTIFICATION: TECHNOLOGY IMPACT MATRIX (TIM) 


Once the compatibility matrix is established, the potential system and sub-system 
level impacts of each technology need to be determined. The impacts must include benefits 
and degradations for an objective assessment. 

Based on the probabilistic nature and issues regarding technological developments 
described above, a Technology Impact Matrix (TIM) is formed for the technologies 
identified in the Morphological Matrix. Recall that the impact in a synthesis/sizing tool is 
simulated via changes in disciplinary technical metrics, "k" factors. Consequently, the 
impact of a technology can be defined by a technical "k" factor vector whose elements 
consist of the benefits and penalties associated with a specific technology. Each element of 
the vector has an estimated impact value and an associated distribution based on the 
technology's TRL. It should be noted that the impact value in the TIM is the ’’theoretical 
limit". With this impact value, the technological uncertainty, or distribution associated with 
a given "k" factor, is defined as a function of TRL and impact value. Not all technologies 
will affect each element of the vector, but the vector must capture all technologies. 



Technology Identification : 

TIM: Technology Impact Matrix 


The technical metric impact shown 
assumes that the impact values are 
achieved when the technology is fully 
matured, i.e. TRL=9 
This assumes that the technology can 
meet the anticipated benefit on 
schedule and within budget. 

Impact values determined through 
literature review and supplied by 
NASA Langley 


: -6» sig- 
ht: +m 

~o i~~ 

~n% 


scs defined the limits for the 
odels of the metrics as a 
M&S Environment 


The sum of .vdtK lions and iiu 
technology space to create met 
function of “k" factors within 
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TECHNOLOGY EVALUATION 


Next, the technologies identified are applied to the vehicle concept and evaluated. The 
evaluation provides data and information to the decision-maker whereby selection of the 
proper mix of technologies may be performed. Yet, the search for the mix that will satisfy 
the customer requirements is dominated by the "curse of dimensionality". Depending on the 
number of technologies (n) considered, the combinatorial problem can be enormous. If all 
technologies were physically compatible, then 2“ combinations would exist considering an 
"on" or "off" condition. In addition, the technology "k" factor vector that influences a 
vehicle is probabilistic and a CDF must be generated for each combination, further 
complicating the evaluation due to the “Curse of Uncertainty”. For example, to estimate the 
impact of uncertainty of a technology combination, a Monte Carlo Simulation of 10,000 
random cases are needed. Hence, I0,000*(2n) combinations would need to be evaluated. If 
the computational expense of the analysis is acceptable, a full -factorial probabilistic 
investigation could ensue. Yet, if the computational expense is too high, an alternate 
evaluation method is needed. A potential method for technology down select is a genetic 
algorithm formulation so as to obtain a more manageable set of alternatives for further 
investigation. 


Technology Evaluation 

• The identification of the proper mix of technologies for a given system is 
dominated by the curse of dimensionality 

• Curse of Dimensionality : the search space for the mix of technologies which 
will “best” satisfy the system level metrics or attributes can be enormous, 
even assuming only an “on”-”off ’ condition 

- 2 n combinations, where “n” is the number of technologies 

• 11 technologies implies 2048 combinations 

• 20 technologies implies 1,048,576 combinations 

- Computational expense of the analysis is the primary driver 

• manageable-, full factorial investigation with metamodel representations 

• unmanageable: genetic algorithms or alternative search algorithms 

• Curse of Uncertainty . Uncertain nature of technologies further complicates 
the evaluation since a probabilistic analysis is needed to evaluate each of the 
2 n combinations 


Dr. Dimitri Mavris 
Georgia Institute of Technology 
Atlanta, GA 30332-0150 
wwtv.asdl.aatech.edu 



Figure 32 


164 





Technology Evaluation: 

”K” Factor Mapping 

5 If the technologies considered can be represented with “k” factors, a metamodel 
representation of the system metrics can be used. 

* For each row in the TIM, the benefits are summed and the penalties are 
summed to bound a given “k” factor such that the ranges are defined for the use 
of a metamodel representation. The ranges define the technology space limits. 


Technical Metric "K" Factor Elements 

Sum of 
reductions 

Sum of 
increases 

K Factor 1 

0 

+10 

K Factor 2 

-40 

0 

K Factor 3 

| -20 

+25 | 


Range of “fc” factor established from TIM 


Response =/ (k,, k 2 ,...„ k n ) as obtained from an application of RSM to acquire a 
second order equation of the form: 


Dr. Dimi+ri Movris 
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TIF ENVIRONMENT*: VISUALIZATION OF THE TECHNOLOGY MAPPING 


Visualization of the influence of the impact factors is performed in the prediction 
profile feature of the JMP statistical package. The information obtained here is similar to that 
of the design space investigation. 


TIF Environment*: 

Visualization of the Technology Mapping 


| LdgFL 


Vapp 


• Identify code 
fidelity needed 
to model a 
technology 

• Impact of 
degradation of 
a technology 
over the life of 
die system 

• Forecasting 
environment if 
no specific 
technologies 
were in mind 

•TIF is a by- 
product of 
applying TIES. 
Bounds of the 
“k” factors are 
purposefully 
determined, a 
pure TIF 
application, 
bounds are 
arbitrary 
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Technology Evaluation 


1 Deterministic evaluation: 

- For a given technology combination, the vector elements that describe the 
technologies are summed and inserted into the metric metamodels and the 
equation evaluated 

- The result is a “theoretical” value of the metric due to the impact of the 
technology combination 

- Advantage: quick assessment and initial insight to technology impacts 

1 Probabilistic evaluation: 

- Define each technology vector element as a function of TRL which will result in 
a distribution for each “k” factor 

- The metric metamodel is evaluated as in the deterministic case except that it is 
repeated numerous times for a given combination to simulate the uncertainty 

- The result is in the form of a CDF for each metric 

-Advantage: realistic assessment of the impact of technological uncertainty 

Assumption: the impact of the technologies are additive 


Or, Dimitri Mavris 
Georgia Institute of Technology 
Atlanta, 6A 30332-0150 
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Technology Evaluation: 

Full Factorial Investigation at “ Theoretical ” Impact 

Case T1 

T2 T3 T„ 

Metric 

i Metric i Metric; 

1 -1 

2 -1 

-1 -1 -1 

1 -1 1 

# 

# 

# # 

# # 

3 -1 

-1 -1 1 

# 

# # 

r*CD ■ 

; O] 1 

| 

# 

* d>] 

Evaluations of Metric RSEs if all 
technologies are compatible 

“1” implies technology applied 
“-1” implies no technology 

Metric value is determined with the RSEs based 
on the technology impact vectors from the TIM 


Dr. Dimitri Mcivris 
Georgia Institute of Technology. 
Atlanta, GA 30332-0150 
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Technology Evaluation: 

Sample of Probabilistic Assessment 

Each technology vector was defined as a function of TRL so as to determine the shape 

distributions of the “k” factors Tl=f(MI.M, ) T2=f(»ffll , ), etc.. 

A Monte Carlo Simulation was executed on the response equations to determine the resulting 
distribution data ‘"*t 1/7''”“' 


T1,T2, T1+T2.. 


Sj stem C ritcria RSEs 
. .for eacb technology : 1 
con*ination V 


The addition of more technologies may move the mean of the response depending on whether 
the technology improves or degrades a given metric. If there is no influence from a given 
technology, the mean will not be affected but the variance may increase 
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$/RPM Frequency Comparison 
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Dr. Dimitri Movris 

Georgia Institute of Technology 


8.501E+5 9.809E-2 


Technologies : (all at TRL=3) 

T2: Composite Fuselage T4: HLFC T5: Environmental Engines 
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Effectiveness Parameter 
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Figure 41 



Figure 42 
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Technology Selection: 

Option 3: Resource Allocation 



Percent Reduction in SLN from Conventional Configuration Percent Reduction in S/RPM from Conventional Configuration 


Technologies: 

Tl: Composite Wing 

T2: Composite Fuselage 

T3: Circulation Control 

T4: HLFC 

T5: Environmental Engines T6: Flight Deck 

T7: Propulsion Materials 

T8: ISSA Structures 

T9: Smart Wing 

T10: Active Flow Control 

T 11 : Active Noise Control 


Decision-maker may readily identify a technology’s influence on a given system metric 
and compare the performance influence and the economic influence. 

Compare confidence level to target reduction needed for feasibility 

Identify the contribution of individual technologies to optimally direct resources 


or. oimitri ifa.ris * provides sufficient SLN reduction, but extreme degradation to $/RPM 

Georgia Institute of Technology 
Atlanta, 6A 30332-0150 

www.03cll.gatech.adtj 
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Figure 43 


Conclusions 




• A comprehensive, robust, and structured process for the 
systematic down-select of technologies for the problem at hand. 

• Process provides valuable insight to the decision-maker in the 
conceptual phases of design to optimally direct program 
resources. 

• No one technology combination can be considered the universal 
solution since the selection process is riddled with subjectivity. 
TIES allows for a quantification and tracking of the 
subjectivity. 

• Future work will focus on: 

- Incorporating schedule and budget issues regarding the development of 
technologies and quantifying the impact on a system 

- Create a recipe for TIES based on generic problem that may be 
encountered during a design process 

44 
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Nondeterministic Approaches 
Potential for Future Aircraft Support 


Dr. James M. Norton 
Lockheed Martin Aeronautics 
Fort Worth, TX 76101 
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USAGE OF NONDETERMINISTIC MODELS 


In the current aerospace environment, which poses continuing challenges to “do more 
with less” the adverse effects of uncertainty on decision making must be recognized and 
minimized. Nondeterministic approaches are predictive tools that permit uncertainties to be 
quantified, by a variety of models, providing decision makers a sound technical basis for 
trade-offs among performance, schedule and cost. The illustrative examples show a broad 
scope for potential applications that don’t require specialized software or advanced statistical 
skills. 


LOCKHEED MART I » ^ 

Usage of Nondeterministic Models 

• Provides formal methodology to quantify the extent and 
consequences of uncertainty inherent in engineering properties / 
characteristics 

• Provides an alternative to “Worst Case” stacking often utilized 
with deterministic methods 



• Concepts will be illustrated by three aircraft support related 
applications 

8/130001 Janes M. Norton 


Figure 1 
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ASSESSING THE EFFECTIVENESS OF TIME CHANGE INTERVALS 


One method of avoiding equipment failures is to remove the equipment from service 
at fixed flight hour intervals. Although failures are avoided, some remaining equipment life 
is lost and the total number of maintenance actions is increased. Analysis of various 
conditional life distributions can help decision makers determine the appropriate policy. 


Assessing the Effectiveness of Time 
Change Intervals 

• Equipment whose life exhibits wear-out characteristics can 
be removed from service at fixed flight hour intervals 
resulting in: 

- Reduced failures 

- Increased maintenance actions 

- Loss of potential remaining equipment life 

• Analysis of the Nondeterministic equipment life 
distribution can help decision makers determine the 
appropriate policy. 

8/13/2001 James M. Norton 


Figure 2 
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ASSESSING THE EFFECTIVENESS OF TIME CHANGE INTERVALS 


Suppose a generator has a Weibull life distribution with mean = 800 flight hours and 
a shape parameter =1.1. A sketch of the life and conditional life distributions may provide 
some preliminary insight as to the effectiveness of a time change after X hours of service. 
The risk of failure prior to the time change, as well as the average remaining life lost, is 
easily computed as shown from common spreadsheet functions. 


LOCKHEED MARTIN 

Assessing the Effectiveness of Time 
Change Intervals 

An example - 

Suppose a generator has a Weibull life distribution with mean = 800 flight hours. A literature 
source suggests a Weibull shape parameter = 1.1. We wish to evaluate the effectiveness of 
removing and replacing the generator every X flight hours. 



_ ( , X N Sha P e h 

Probability failure prior to X = 1 - EXP y- Vscale/ / 

Average remaining life = 

Scale • EXP (K1 + GAMMALN (K2)) • (1 - GAMMAD1ST (K1 , K2, 1 , TRUE)) - X 
/ v \ shape 

Where Ki = sca ]" e "y , K2 = 1 + 1/shape, GAMMALN ( ) = Excel Supplied Function 

GAMMADIST ( ) = Excel Supplied Function 


8 / 13/2001 


James M. Norton 


Figure 3 
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ASSESSING THE EFFECTIVENESS OF TIME CHANGE INTERVALS 


The results in the table show modest reductions in failures from time changes (shape 
=1.1), which need to be assessed in conjunction with the consequences of failure. The likely 
results and sensitivity of results to variations in model parameters can be examined before 
any data is available. Statistical estimation methods are used to draw inferences about model 
parameters from data. 


LOCKHEED MARTIN 


Assessing the Effectiveness of Time 
Change Intervals 


WEIBULL SHAPE 


Time Change 
Interval 

1.1 

1.3 

1.5 


20 

20 

20 

400 

8.8 

7.0 

5.7 


14851 

13179 

12032 


10 

10 

10 

800 

9.3 

8.2 

7.4 


7160 

5918 

5044 


0 

0 

0 

No Time 




Change 

10 

10 

10 


0 

0 

0 


# Time Changes 

# Failures (Weibull 

Renewal) 

Unused Life 

# Time Changes 

# Failures (Weibull 

Renewal) 

Unused Life 

#Time Changes 

# Failures (Weibull 

Renewal) 

Unused Life 


TABLE ENTRIES ARE AVERAGES FOR 8000 FH A/C LIFE. 





A SUFFICIENTLY LARGE PROBABILITY OF A BRAKE FIRE 


Frequently during landing / taxi military aircraft experience significant increases in 
brake temperature which, if coupled with hardware failures in the fuel system, could result in 
a fire during refueling. A nondeterministic model can estimate the frequency of fires, to help 
determine what (if any) interventions are needed. 



A Sufficiently Large Probability of a 
Brake Fire May Require New Support 
Equipment or Design Changes 


Probability t 

Temperature 
Hot Enough 
Ignite the Fi 


James M. Norton 
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Figure 5 
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CONDITIONAL PROBABILITY BRAKE TEMP 


Key factors that determine brake temperature are defined during discussions with 
technical experts. Each factor is characterized by a probability distribution, derived from 
data and/or inputs from experts. The individual distributions are combined, reflecting any 
significant correlations, to form the brake temperature distribution. Live testing under 
realistic environmental conditions results in an estimated fuel ignition distribution that is 
combined with the brake temperature to quantify risk. 


LOCKHEED MARTI 




Conditional Probability Brake Temp is 
Hot Enough to Ignite Fuel 



James M. Norton 


Figure 6 
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PRICING REPAIRS FOR A FIXED TIME PERIOD 


To help customers plan for the costs associated with repair of equipment failures in 
their aircraft fleet, a contractor may offer to provide repairs for a fixed time period at a pre- 
specified price. One NDA approach to help establish an equable price is to model cost as a 
compound random variable. 


LOCKHEED MARTI 




Pricing Repairs for a Fixed Time Period 


• All Equipment Failures that Occur During a Fixed Time 
Period will be Repaired for a Pre- Specified Price 

• One NDA Approach to Help Establish an Equable Price is 
to Model Cost as a Compound Random Variable: 

Total Cost = Cj + C 2 + — + C n where 

Ci = Random Variable 
Cost of i* Repair 
N = Random Variable 
Representing 
Number of Repairs 

in the Defined 
Time Period 

8/13/2001 James M. Norton 


Figure 7 




PRICING REPAIRS FOR A FIXED TIME PERIOD 


Frequently, the number of failures to be repaired in a fixed time period can be 
characterized by a Poisson distribution. Repair cost means and variances are then utilized to 
compute the mean and variance of total repair cost. The risk that the cost of the repairs will 
exceed the price charged can be assessed after a distributional form for cost is selected. 
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DIVERSE SUPPORT RELATED APPLICATIONS BENEFIT FROM 
NONDETERMINISTIC APPROACHES 


The use of nondeterministic methods for support-related applications has a long 
history, with many companies establishing formal specialty groups that concentrate on 
application areas, such as safety, reliability, maintainability, and logistics support. Basic 
models in the specialty area are generally well known and have developed some commonly 
accepted standards for use. The rapidly increasing capability to collect, store, process, and 
share data will reveal new relationships among variables, leading to the development of more 
sophisticated nondeterministic methodology. 


L O 



Diverse Support Related Applications Benefit 
from Nondeterministic Approaches 


• Tech Order Changes (Often Inspection Related) 

• Rates of Damage Accumulation 

• Spares Requirements 

• Warranty Provisions 

• Repair Turnaround Capability 

• Diagnostics / Health Monitoring 


3/13/20Q1 
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KEYS FOR SUCCESSFUL IMPLEMENTATION 


As advances in nondeterministic methods are achieved, some basic keys for 
successful implementation remain unchanged. Although the items listed are fairly obvious 
and not exhaustive, they are easy to forget. 



Keys for Successful Implementation 


• Establish clear objectives / expectations for model 
results 

• Develop model structure with guidance from experts 
in affected areas 

• Collect sufficient reference information / data for 
credible model parameter estimates 

• Summarize / communicate results that are technically 
rigorous, yet readily understood 

- Avoid technical jargon 

- Delineate assumptions 

- Graphically depict results, where possible 

S/13/2001 Janes M. Norton 


Figure 10 
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ANSYS Probabilistic Design System 
Exploring Randomness and Scatter Reveals A Simple Truth 


Dr. Stefan Reh 

Team Leader Probabilistic Design 
ANSYS Inc. 
Canonsburg, PA 15317 
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ANSYS PROBABILISTIC DESIGN SYSTEM 


In December 2000 ANSYS Inc. has released the version 5.7 of its ANSYS Finite- 
Element program. This release (and any following ones) has a probabilistic design system 
integrated into it. This tool enables users to take uncertainties of their Finite-Element model 
input parameters into account. The probabilistic approach is more realistic and closer to 
reality than the purely deterministic approach, which tends to ignore uncertainties. Using the 
ANSYS Probabilistic Design System (ANSYS/PDS) users can quantify the quality and the 
reliability of their products and consequently make informed decision on how to improve 
their products with respect to quality and reliability. 


ANSYS Probabilistic Design System 




Probabilistic Design: Bringing Engineering closer to REALITY! 

_____ AN 

Probabilistic Analysis of Gas Turbine Engines using the ANSYS Probabilistic Design System 


Exploring Randomness and Scatter Reveals a Simple Truth 
ITS A PART OF REALITY - EVERYWHERE 


Figure 1 
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PROBABILISTIC DESIGN SYSTEM: INTRODUCTION 


If we analyze a component (for example a turbine blade like those shown here), then 
we start out with the fact that the component is described by a certain set of input parameters, 
namely material properties, geometric extensions of the component and boundary conditions 
that describe how the component is loaded and where and how it is fixed. Then the 
component is analyzed, and as result we can look at its deformation and review the stresses 
and strains. In addition, we can assess its fatigue lifetime or its creep behavior and such. 
Probabilistic Design is based on the fact that all input parameters are subjected to scatter. 
Take, for example material, properties. If you measure a particular material property then you 
will observe measurement values that are different from specimen to specimen. Also, the 
geometric extensions of a component can only be manufactured within certain tolerances. To 
strive for perfection is physically not possible and even trying to get close to perfection is not 
reasonable in financial terms. Also loads and boundary conditions are subjected to scatter, 
i.e. there are some uncertain influences that we have to accept and live with. As a direct 
consequence we have to face the fact that the output parameters are subjected to scatter as 
well, i.e. they are uncertain as well. ANSYS has developed a probabilistic design system that 
can take the randomness of such input parameters into account and provide the necessary 
conclusions. 


ANSYS Probabilistic Design System: 
Introduction 


Component 

description 


-^Material 
-^Geometry 
^ Loads 

-^Boundary Condition 


Itattfe* 




Deformations 

-^Stresses 

■^Lifetime 

(LCF,...) 


As a consequence of the 
uncertainties of the input 
parameter there will be also 
uncertainties of the results 


wife 


Probabilistic Analysis of Gas Turbine Engines using the ANSYS Probabilistic Design System 


AN 


Figure 2 
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PROBABILISTIC DESIGN SYSTEM: FEATURES 


The ANSYS probabilistic design system is available at no charge for customers 
having an ANSYS license. It is automatically a part of any ANSYS product. It works with 
any ANSYS model no matter what the underlying physics is. It allows for a large number of 
random input and output parameters. The random input parameters can be correlated in order 
to address random fields. Two widely used and very robust probabilistic methods have been 
implemented, namely then Monte Carlo Simulation method and the Response Surface 
Method. Both are available with various sampling techniques. In order to fit response 
surfaces, the ANSYS/PDS offers several sophisticated regression analysis techniques. 
Included are techniques to apply transformation functions for cases where a quadratic 
response surface is not sufficient. Also filtering mechanisms can be used that will filter out 
insignificant terms in the regression model in order to avoid the “over-fitting” problem. To 
reduce the wall clock time of a probabilistic analysis, the PDS comes with a tool that will 
automatically distribute the various jobs in a heterogeneous network of computers. This 
includes the capability to submit a job that has failed due to CPU or network problems to 
another CPU. Naturally, the ANSYS/PDS offers a great variety of tools to visualize the 
probabilistic results, such as histogram plots, cumulative distribution plots, scatter plots, 
sensitivity plots and so on. 


ANSYS Probabilistic Design System: 

Features 

• The ANSYS/PDS is FREE for every ANSYS customer 

• It works with any ANSYS model (static, dynamic, linear, non-linear, thermal, 
Structural, Electro-magnetic, CFD ...) 

• It allows for a large number of random input and output parameters 

• It has 1 0 statistical distributions for random input variables 

• The random input variables can be correlated 

• Probabilistic methods: 

Monte Carlo • Direct & Latin Hypercube Sampling 
Response Surface - Central Composite & Box-Behnken Designs 

• Sophisticated regression analysis capabilities for response surface fitting 
(automatic transformation functions for a “more than quadratic” fit, automatic 
filtering of insignificant regression terms to avoid “over-fitting” problem) 

• Use of distributed, parallel computing techniques for drastically reduced wall clock 
time of the analysis 

• Comprehensive probabilistic results (convergence plots, histogram, probabilities, 
scatter plots, sensitivities, ...) 

• State-of-the art statistical procedures to analyze and visualize probabilistic results 

Probabilistic Analysis of Gas Turbine Engines using the ANSYS Probabilistic Design System 


Figure 3 
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PROBABILISTIC DESIGN SYSTEM: CUSTOMER BASE 


As many people know, ANSYS is widely used in the industry, and here you see only 
a few of our customers who gave us permission to show their company logo. The ANSYS 
Probabilistic Design System is an integral part of ANSYS 5.7. More than 35 companies are 
already using it worldwide. 


ANSYS Probabilistic Design System: 
Customer Base 



ANSYS Customer Base 


• All “Top 10” Fortune 
100 Industrial 
companies 

• 73 of the Fortune 100 
Industrial companies 

• Over 5,700 commercial 
companies 

• Over 40,000 commercial 
customer seats 

• Over 100,000 university 
licenses 

Probabilistic Design 

• Available in ANSYS 5.7 

• Used by 35 companies 
worldwide 
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Figure 4 


190 












RELIABILITY OF COMPONENTS: EXAMPLE TURBINE BLADE 


As an example for the application of the ANSYS probabilistic design system, the 
probabilistic analysis of a turbine blade is shown here. The example is a cooled (hollow) 
rotating blade. The probabilistic analysis includes the randomness of a total of 17 input 
parameters. For example, the blades are manufactured by precision casting. During casting of 
the blade a slight shift of the core that makes up the hollow cavity can occur. This core shift 
makes the wall of blade thinner on one side and thicker on the other. Also there is an 
oxidation protection coating on the hot gas surface of the blade. The thickness of the coating 
is not an exact value after it is applied, but variations from the targeted thickness may appear. 
It is not necessary to explain all random input parameters as listed here. Suffice it to say that 
the random input parameters are from all categories, namely geometry, material and loads. 
Also it should be emphasized that various different statistical distribution functions can be 
applied to describe and quantify the randomness of the input parameters, such as the 
Gaussian distribution, the uniform distribution or the lognormal distribution (the 
ANSYS/PDS has many more). This Finite-Element model has about 60,000 elements and 
180,000 nodes. One single analysis run includes a thermal analysis to evaluate the 
temperature field (shown here) and a structural analysis to evaluate the thermo-mechanical 
stresses. Based on these results the low cycle fatigue lifetime (LCF), the creep lifetime and 
the time until the oxidation protection layer has been eroded through is calculated. One 
complete analysis as described takes about 2 hours. 


Reliability of Gas Turbine Components: 
Example Turbine Blade 



17 Random Variables for input variables 
Geometry parameters 

Cooling channel shift (Circumference) UNIF(-0. 6,0.6) 
Cooling channel shift (Axial) UNIF(-0. 6,0.6) 

Thickness of oxidation protection LOG(0.3,0.03) 

Material parameters 

Young’s Modulus (*) NORM(1 .0,0.04) 

Density (*) NORM(1 .0,0.05) 

Therm. Expansion (*) NORM(1 .0,0.05) 

Heat conduction (*) NORM(1 .0,0.05) 

Heat capacity (*) NORM(1 .0,0.04) 

Oxidation depletion rate (*) LOG(1 .0,0.05) 

Strength related material parameters 
LCF curve (*) LOG(1 .0,0.15) 

Creep rupture curve (*) LOG(1 .0,0.10) 


Thermal Boundary Conditions 

• Hot gas temperature 

• Hot gas heat transfer coefficient (*) 

• Cooling air temperature 

• Cooling air heat transfer coeff. (*) 

• Hot gas mass flow (*) 

• Cooling air mass flow (*) 


NORM(0. 0,25.0) 
LOG(1 .0,0.2) 
NORM(0.0,10.0) 
LOG(1 .0,0.1) 
NORM(1. 0,0.03) 
NORM(1. 0,0.05) 



Thermo-mechanical analysis 
60’000 Elements (quadratic) 
1 8CT000 Nodes 


Temperatures 


3 Output parameters 

• LCF lifetime 

• Creep lifetime 

• Oxidation lifetime 


(*) Factor relative to nominal value or curve 


2 h of CPU time per analysis 

AN 
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Figure 5 
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RELIABILITY OF COMPONENTS: FAILURE PROBABILITY 

OF TURBINE BLADE 


Crucial in today's business environment is the development of reliable products. Only 
reliable products keep the occurrence of premature failures (a failure that happens before the 
end of the warranty period) at an acceptable level or avoid such failures completely. The 
most important measure for a reliable product is a low failure probability. As a result of the 
probabilistic analysis in this diagram, the probability of a failure due to one of the three 
failure modes (LCF, creep, oxidation) is plotted versus the operation time in years. A 
particular failure probability can be derived from this plot by choosing a value on the X-axis 
for the operation time (i.e. the time how long the blade is supposed to be in service) and then 
going up to the probability curves related to the failure modes and reading the probability on 
the Y-Axis. In this diagram, the results calculated with the “response surface method” are 
compared with the results gained from 500 Monte Carlo simulations. In this example, the 
Monte Carlo Simulation results provide benchmark values the “response surface method” 
results should agree with for the failure probability ranging from 2% to 98%. Obviously, 
there is a very good agreement between the results of the two methods in this probability 
range. 


Reliability of Gas Turbine Components: 
Failure Probability of Turbine Blade 



Failure Probability of Individual Failure Modes 

99.999 


r _g9.99 
£1 99.9 



100 1000 10000 

Operation Time [years] 


100000 
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Figure 6 
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RELIABILITY OF COMPONENTS: SENSITIVITIES FOR TURBINE BLADE 


Probabilistic methods also automatically deliver probabilistic sensitivities. These 
sensitivities describe how much the scatter or the failure probability of a particular random 
output parameter (shown here is the LCF lifetime) is affected by the scatter of the individual 
random input variables. The ANSYS/PDS sorts the input variables into two groups - the 
significant and the insignificant ones. Then the significant input variables are ranked by the 
importance and plotted. These probabilistic sensitivities provide highly valuable information 
in many ways. If the resulting failure probability is too high, then we need to improve the 
design in order to achieve an acceptable level. The sensitivities clearly indicate which input 
variables are the drivers of the high failure probability. Hence, the input parameters must be 
tackled in the order of their importance. There is no point in focusing on unimportant 
parameters. Sometimes the scatter of some input parameters are just estimated based on no or 
very little measurement data. If these parameters turn out to be very important for the 
reliability of the design, this indicates that lab tests should be done to collect more data about 
that input parameter. If the current design is sufficient, i.e. has an acceptably low failure 
probability, and then there is typically the need to save money without sacrificing the 
achieved reliability. In this case, the manufacturing requirements for the input parameters can 
be relaxed and a possibly coarser or cheaper manufacturing process can be chosen, or the 
quality assurance requirements for those parameters can be relaxed. This typically leads to 
huge savings in the manufacturing process. 


Reliability of Gas Turbine Components: 
Sensitivities for Turbine Blade 



Sensitivities of the output parameters 
with respect to the random input variables 



Rank-Order Correlation Sensitivities 

Result Set LHSJRUN 


Significant: 
m DEL_LCFLIFE 
j£ FCT_HTC_MOT 
M FCTJTHERMEXP 
m DEL_TEMP_HOr 
m FCT_¥OUJTGS 
FCTJEHERMCCMB 
a FCTJOTSITV 


significance level: 

omm 


Output Parameter LCF_L!FE 


— AN 
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RELIABILITY OF COMPONENTS: EXAMPLE TURBINE STAGE 


Products, of course, usually consist of many components and not just one. Therefore, 
to assess the reliability and quality of a product, it is important to evaluate the system 
reliability of all parts the product is made of. In the example here, we consider an entire 
turbine stage consisting of 100 turbine blades on the circumference plus the turbine disk on 
which these blades are mounted. For the turbine blades the same input parameters are 
considered as uncertain input variables as described above. For the turbine disk, a separate 
finite-element model has been built and evaluated for its fracture mechanical lifetime. The 
fracture mechanical lifetime is driven by the existence of a crack in the disk center that is 
small enough to just be overlooked by non-destructive inspection. The crack will grow in 
size due to the cyclic loading of the disk by the start-up and shutdown cycles. To model the 
crack growth a simple Paris law has been used. The crack growth is governed by the initial 
crack size, the fracture toughness and the crack growth parameters of the Paris law, all of 
which are included in the probabilistic analysis as random input parameters. 



Reliability of Gas Turbine Systems: 
Example Turbine Stage 



Turbine Stage 

100 Turbine blades on circumference 
1 Turbine disk 

Random input variables for turbine blades 

• Geometry parameters (as described above) 

• Material parameters (as described above) 

• Strength parameters (as described above) 

Random input variables for turbine disk 

Cyclic crack growth of an existing crack in disk center that 
is just not detectable in non-destructive inspection 

~ = AAKf = A(A^VJy)" _S«2U K lm = K , c 

• Initial crack size a init 

• Fracture toughness^ 

• Crack growth parameters A and n 

Random input variables for entire stage 

• Thermal boundary conditions (as described above) 
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Figure 8 
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RELIABILITY OF COMPONENTS: COMPONENTS OF THE TURBINE STAGE 


In the top half, the results for the individual blades are shown. This is the same 
picture as already shown before. Shown in the bottom half is the result for the turbine disk. 
The Finite-Element model of the disk shows the stress field in the disk. The failure 
probabilities due to a fracture mechanical failure of the disk is shown in the bottom right 
diagram. 


Reliability of Gas Turbine Systems: 
Components of the Turbine Stage 




Entire Turbine Stage 


Temperatures 


100 Turbine Blade on 
Circumference 


1 Turbine Disk 


(*) Grooves for blade 
root attachment are 
not included in model 

potential crack origin 


Failure Probability of Individual Failure Modes 



Failure Probability of Turbine Disk 



Operation Time [years] 

—AIM 
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RELIABILITY OF COMPONENTS: COMPONENTS OF THE TURBINE STAGE 


This diagram shows the overall failure probability results for the entire turbine stage 
plotted versus the operation time in years. The green curve is the probability that the disk 
fails due to crack growth. The brown curve is the probability that any of the 100 blades fails 
due to oxidation. The pink curve represents the probability that any of the 100 blades fail due 
to creep. Analogously, the blue curve is the probability that any of the 100 blades fail due to 
low cycle fatigue. The red curve is the probability that the turbine stage fails due to any of 
these component events, i.e. failure of the disk OR failure of any blade due to any failure 
mode. Obviously in this example here, the turbine stage as a system is driven only by a 
failure of the blades due to creep or LCF. 


Reliability of Gas Turbine Systems: 
Failure Probability for Turbine Stage 


Probabilistic Analysis of the Turbine Stage 
using the Response Surface Method 
Failure Probability of Entire Turbine Stage 
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RESEARCH OBJECTIVE 


Deterministic optimization does not account for neither the inherent variability of the 
operating conditions nor the model uncertainties. The objective of this research is to adapt 
the existing optimization techniques to automatically include these effects. This results in 
more robust designs. 


Research Objective 

• Observation: use of mathematical optimization 
techniques frequently leads to designs whose 
performance is very sensitive to small fluctuations 
in the design model parameters. 

• Some of these parameters can be highly variable 
or hard to estimate. 

• Objective: adapt existing optimization techniques 
such that the solution becomes fairly insensitive to 
(minor) fluctuations in some or all of the 
mathematical model parameters. 


Figure 1 





DEFINITION OF ROBUST DESIGN 


Robust optimization results in the design, which performs optimally under the 
variable (or uncertain) operating conditions over the entire lifetime of the design. 

For this computation we assume there are no catastrophic failures; we are dealing 
with everyday fluctuations. This is quite different from reliability computations. 


Reliability Problem Classification 


No engineering 
applications 


Cost-benefit analysis 
Robust design and 
optimization 


Risk analysis 
Reliability -based- 
design & optimization 


Reliability is 
not an issue 



Everyday fluctuations Extreme events 

Frequency of event 


Figure 2 
200 



AIRFOIL GEOMETRY OPTIMIZATION 


Find optimal airfoil geometry, which results in minimum drag Cd over a range of free 
flow Mach numbers M while maintaining a given lift C* = 0.6. We start from a NACA-0012 
airfoil. 

For this example we assume a uniform distribution for the Mach numbers: M □ [0.7, 
0.8]. All Mach numbers within this range are equally likely. The Mach number cannot fall 
outside this interval. 

We solve the inviscid Euler equations using NASA’s FUN2D code, which computes 
analytic derivatives. Far field boundary at 50 chord lengths. 


Design Variables in FUN-2D 



Figure 3 
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OPTIMIZATION FRAMEWORK 


We used the ISIGHT framework, a product of Engineous Software, as the 
optimization engine. The framework provides an interactive high-level programming 
environment to define the optimization problem. It automates the communications between 
the user-provided analysis codes and the pre-programmed optimization routines. 


iSight Optimization Framework 

* Task Manager 

• Process Integration 

- Parser for Input/Output files 

- Control structures 

# Parameter Definition 

- Variables, objectives, constraints 

# Optimization Plan 

• Solution Monitor 

- Tracks all design variables during optimization 


Figure 4 
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PROCESS INTEGRATION 


The data flow during the optimization process is defined in the Process Integrator that 
includes several control structures and statements (IF, WHILE, LOOP). The flow is built up 
using a GUI, or can be implemented directly in the MDOL language. 
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4igrs.dmtat 
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INPUT - OUTPUT 


The built-in optimization routines communicate with customer software using input 
and output files. The Parser commands can be provided directly using the MDOL syntax, or 
can be generated by iSight using the provided GUI. 



Figure 6 
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MODEL PARAMETER DEFINITIONS 


Independent variables, objective functions and constraints are specified in the 
Parameter Definition module. Simple bounds can easily be included for the independent 
design variables. 
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OPTIMIZATION ROUTINES 


Several Optimization routines are built-in. We made use of the Modified Method of 
feasible Directions (CONMIN) and Sequential Linear Programming (based on ADS 
routines). 



ffptiwW.lOfr Platt; 


IMl 


•Optimization FlartfjSLP 
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Select a Step to show details tor; 
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■Sequential Linear Programming 
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RESULTS MONITOR 


All intermediate results of the optimization process are stored in a database. The 
results monitor displays the optimization histories in real-time. Here we show the 
optimization history for a drag-objective. 



Nav^ator-j Spft t < fragi-m j Data j 

% Solution Monitor Views Path; /ump/icast 
Ej© ..yMy^Point/OptimizabcnOr ; 

^^iijRunCaiimer used {0:1 J* 
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DETERMINISTIC OPTIMIZATION STRATEGIES 

We highlight two methods: single-point and multi-point optimization 


Single Design-Point Optimization 

• The design vector d (geometry and angle of 
attack) is the only variable in the objective 

* Fix all other model parameters at their 
design value. We consider only 1 free flow 
Mach number M- M desi ( e.g . average 
Mach number during cruise stage): 

ynin C d (d,M design ) 

subj ect to C,(d, M iesign ) > C ; * 


Figure 10 
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PROBLEMS WITH SINGLE-POINT OPTIMIZATION 


Not clear which point to select as design point. The mean value is not a good choice 
for the design point when the model is highly non-linear. 

Even though we are trying to push out Morv, the highest Mach number (M = 0.8) is 
not necessarily the best design point either. 

The impact of fluctuations of the model parameters (due to either inherent variability 
or model uncertainty) on the response is completely unknown. The optimized design may 
actually perform worse under such “off-design point” operating conditions. 






MULTI POINT OPTIMIZATION 


Attempt to overcome difficulties of single-point optimization by including several 
design operating conditions in the objective function 


Multi-Point Optimization 

• The design vector d (geometry and angle of 
attack) is the only variable in the objective 

• Consider multiple design conditions at selected 
values of the free flow Mach number 

• Objective function is a weighted average of all 
these design conditions 

min y Wl C d (d,M,) 

< deD “ 

1=1 

subject to Ci (d,M t ) > C* for i = l,n 


Figure 12 
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PROBLEMS WITH MULTI POINT OPTIMIZATION 


The resulting drag profile is sensitive to the choice of Mach numbers. It is not clear 
how to decide which Mach numbers to include in the objective. 

What is the appropriate weight for each design condition (i.e. Mach number) in the 
overall linear combination? 

Multiple drag troughs can be observed, one at each sample point. 


Problems with Four-Point Opt. 

Choice of design conditions affects performance 
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STOCHASTIC OPTIMIZATION 

We seek to maximize the expected performance over all possible operating 
conditions. 


Stochastic Optimization 

• Modify the objective to directly incorporate 
the effects of model uncertainties on the 
design performance 

• Highlight 2 methods: 

- Expected Value Optimization 

- Second-Order Approximate Results 


Figure 14 
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STOCHASTIC OPTIMIZATION USING DECISION TREES 


Consider all possible designs di and set up a decision tree. 

Model all uncertain variables using (Joint) Probability Density Functions 

The objective is to minimize the drag over the entire Mach range. 

This shows that the best decision (or design) is the one that minimizes the expected 
value of the drag Q with respect to M. 



Figure 15 
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MATHEMATICAL FORMULATION 


The objective in the optimization process is given by the expected value of the drag 
with respect to all possible operating conditions. We integrate the product of the drag 
function Q and the probability density function of the Mach number 


Mathematical Formulation 

Minimize the expected value of the drag 
over the design lifetime: 

min E m (C d (d,M))= min [ C d (d,M)f M (M)dM 

deD deD JM 

C d is drag function 

d is design vector (geometry, angle of attack) 

M is uncertain parameter (Mach number) 

f M is Probability Density Function of Mach number 


Figure 16 
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APPLICATION TO AIRFOIL PROBLEM 

The design variables are the geometry of the airfoil. The angle of attack is a control 
variable. The angle of attack will be adjusted to achieve the required lift when the Mach 
number fluctuates. Here we only model slowly varying Mach numbers so that the constraint 
remains deterministic. Probabilistic constraints will be considered in a future study. 


Application to Airfoil Problem 

• Integrate over the uncertain parameter M, compute 
the expected value of C d with respect to the free 
flow Mach number M. 

• Minimize this integrated objective with respect to 
the design vector d. 

• Actual flight data are readily incorporated in the 
probability density function f^M) 

min [ C d (d,M)f M (M)dM 

< deD JM 

subject to Ci >c; 


Figure 17 
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SECOND-ORDER SECOND-MOMENT APPROXIMATION 


When the objective function is approximated by a second-order Taylor series 
expansion, an analytic evaluation of the integral can be performed. This leads to a 
deterministic equivalent of the stochastic optimization problem. 


SOSM Approximation 

Approximate objective by second-order Taylor 
series expansion about the mean value of M, and 
evaluate the expectation integral analytically. 

min f C d (d,M)f M (M)dM = 

deD J 
M 


min 

C d (d , M ) + 


deD 

dM 2 

M-M _ 


subject to: C t > C* 


Figure 18 
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EFFECT OF SOSM CORRECTION 


Second-Order information represents curvature of Q-M curve. 

The weighting between drag and design point and curvature depends on the variance 
of the Mach number. 

With SOSM method the drag is not reduced quite as much as for single point design 
but the drag is much less sensitive to variations in the Mach number. The drag trough is 
avoided, no “over-optimization”. 

Second-Order derivatives were computed using finite differencing of first-order 
derivatives. This introduces numerical noise in the derivative and may “confuse” the 
optimization algorithm. 


Comparison with Single Point Opt. 
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EXPECTED VALUE OPTIMIZATION 


Here we use a numerical integration procedure to perform the integration of the 
objective function during each optimization step. This is computationally more expensive 
but also more accurate. 


Direct Evaluation of Integral 

• Evaluate integral directly using a numerical 
integration method. 

• To avoid over-optimization, make sure you 
select different integration points for each 
optimization step. 

•We used 4 point integration with random 
selection of integration points. 


Figure 20 
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ADVANTAGES OF EXPECTED VALUE OPTIMIZATION 


Robust design consistently has smallest expected value (up to accuracy of 
integration). 

There is no need to arbitrarily select design conditions (i.e. Mach numbers) or 
weights any longer because we integrate over the PDF of the operating conditions. 

Drag troughs are reduced and do not occur at integration points any longer. 

Additional model uncertainties can be accounted for as well by extending the 
integration over the uncertain model parameters. 


Comparison with Multi-Point Opt. 

EV-design independent of arbitrary selection of Mach numbers 



Figure 21 
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COMPUTATIONAL EFFORT 


SOSM Method replaces integration with a second-derivative evaluation. If accurate 
higher-order derivatives can be computed, this method results in significant computational 
savings over the full integration. 

SOSM scales linearly with the number of uncertain variables, while integration 
algorithms generally follow a power law. 


Relative Computational Effort 


Optimization 

1 Random 

3 Random 

Method 

Variable 

Variables 

Single-Point 

1 

1 

SOSM(*) 

3 

7 

Expected 
Value (4pts) 

4 

64 


(*) Less if analytic derivatives are available 
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ASSUMPTION OF PDF 


For bounded Mach-intervals a Beta-distribution seems appropriate. The Beta PDF 
can represent a variety of shapes (symmetric or not, bathtub. . For unbounded distributions 
a detailed analysis of the tails is required. 

Here we show several assumptions for the PDF, and will assess their impact on the 
optimal drag-rise curve. 


Various PDFs for Mach Number 

“ — Beta(3,l) Beta (1,3) Beta (5,5) Trunctated Normal 



Figure 23 
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IMPACT OF PDF 


The Expected Value Optimization results automatically reflect the relative 
importance of each of the Mach numbers. Depending on the relative likelihood of the Mach 
number (shown in previous slide), the drag is reduced to a larger or smaller extent. This 
shows that the optimal drag-rise curve depends on the probability density of the Mach 
number. 







FURTHER WORK AND SUMMARY 


In future research we will extend the method to include effects of other uncertainties 
besides the Mach number; preferably using faster integration techniques (adaptive sampling). 
We will also include probabilistic constraints. 


Conclusions 

• Statistical decision theory indicates that 
minimizing the expected drag over the lifetime 
leads to the optimal robust design. This removes 
the arbitrariness from the selection of the design 
conditions and/or weights, which is found in 
multi-point optimization. 

• SOSM shows considerable improvement in the 
robustness of the design compared to single-point. 

• The SOSM analytical approximation shows that, 
at the mean Mach number, the first-order 
sensitivity does not affect the expected value of 
the design. 


Figure 25 
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AN OVERVIEW 


The initial version of the Robust Design Computational System (RDCS) was 
developed as a cooperative effort between DARPA, The Boeing Company, The Ford Motor 
Company and MacNeal Schwendler Corporation during 1996 -1999. Since then the product 
is continually being enhanced using BOEING internal funds. The RDCS product is being 
used internally within Boeing at several sites. A commercial version of the product called 
MSC.Robust Design is being marketed by MSC software as part of engineering service 
engagements. 


An Overview of the Robust Design Computational System 

(RDCS) 

A Collection of Tools To Enable Low Risk Designs 

Dr. Raj Rajagopal 
Technical Fellow 

Boeing - Rocketdyne Propulsion and Power 
raj . rajagopal @ boeing.com 


Training Workshop on Nondeterministic Approaches and Their 
Potential for Aerospace Systems 
May 30-31,2001 
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FUNDAMENTALS THAT DROVE THE RDCS DEVELOPMENT - THE LAUNCH 

MARKET BUSINESS CASE 


In the following few slides I set the background reasons why the RDCS system was 
developed with its implemented features. It is based on a sound business case. 

Before the 80’s the DOD and NASA programs the emphasis was on performance. But 
the budgetary pressures, the global competition and meeting the increasing demand by 
product customers for performance at an affordable cost with out sacrificing reliability 
became a paramount goal. The engineers at the working level are now schooled, lectured and 
admonished to be sensitive and responsive to total life cycle cost issues. The difference can 
be between having a program and not having one. Experts might argue on the order of faster, 
cheaper and better but all three elements are needed. 








LOW-COST DEVELOPMENT CAPABILITY NEEDED 


Consider the development cost of major products in different industries such as rocket 
propulsion, automobile and jet engines. The schedule and cost are dominated by the “Test 
Fail Fix” design methodology. That is, there is heavy reliance on test to meet the high 
reliability and performance goals. This has been a proven approach that has produced reliable 
products, but the cost is not consistent with the new accelerated schedule and cost goals. So 
how what can we do differently? 
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FUNDAMENTALS THAT DROVE THE RDCS 
DEVELOPMENT - AEROSPACE DESIGN SHOP 


Before we design a system, we need to look at the environment at which the DOD 
and NASA product design teams are operating. First of all we cannot afford to fail as the 
consequence of failure (cost) is very high. We live in a glass house. The rigors of low weight 
and high performance demand the use of cutting edge technologies. There is tremendous 
progress on the accuracy of numerical model prediction. This when combined with the 
computer revolution is providing us with an unprecedented opportunity to design the 
products right the first time. Any design framework that has the potential to provide a 
solution to this problem must consider the elements listed on this slide and design the system 
accordingly. 


Fundamentals that Drove the RDCS Development 

Aerospace Design Shop 


• Consequence of failure high 

• risk = f(probability of failure, cost of failure) 

• Cutting edge technologies 

• Compute intensive analysis 

• Limited quantity or one of a kind hardware 

• Weight critical 

• Geographically dispersed design teams 

Rocketdyne a 

Propulsion & Power 
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FUNDAMENTALS THAT DROVE THE RDCS DEVELOPMENT - AUTOMATION 

VS. PROCESS IMPROVEMENT 


There is a big difference between automation and a real design process improvement. 
Automation makes execution of the existing design processes faster. If a failure occurs 
automation will certainly help recover from the failure faster and cheaper, but what we want 
to aim is to have no failure in the first place. That requires changes and improvement in 
design quality and design process. It requires a paradigm shift. 

Quality improvement in space products with an acceptable cost and reliability can not 
achieved with out considering variability in a very fundamental way in the design process. 



Fundamentals that Drove the RDCS Development 

Automation Vs Process Improvement 

• Automation is normally speeding up the design process 

• Traditional CAE 

• Partial but not the full answer 


• Improvement in design quality requires a more 
fundamental change 

• Quality is meeting contractual/customer expectations of 
the product performance consistently every time 

• It is impossible to think quality improvement without 
considering variability 


• Reliability is a significant attribute contributing to the 
quality 
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FUNDAMENTALS THAT DROVE THE RDCS 
DEVELOPMENT -A QUALITY SCENARIO 


Robustness of a product should not be considered as a product with high margins. 
The reality of high performance and weight critical parts do not allow us that luxury very 
often. The robustness is defined here as a designed in quality of the product that makes it 
performance insensitive to variations. 

All non-trivial design activities are trade off exercises. Consider the cartoon in this 
slide of one performance Vs one design variable scenario. The design problem is to choose a 
nominal design point in the presence of uncertainties or variations. The robust design choice 
cannot be made unless the variations are considered and then an intelligent design decision 
can be made. When we say we need to optimize the product performance, we need to set the 
objective such as “This product shall met the performance goal 99.99% of the time” which is 
a probabilistic optimization statement. 


Fundamentals that Drove the RDCS Development 

A Quality Scenario 



Lower performance operating 
point but relatively insensitive to 
variations 


Input Variable 
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FUNDAMENTALS THAT DROVE THE RDCS DEVELOPMENT - MULTI- 
DISCIPLINARY APPROACH A MUST 


Earlier efforts in the 80’ s were concentrated on single discipline approach to 
reliability estimation such as structural reliability. However, the source of component 
unreliability is driven from many sources such as manufacturing, process, inspection process, 
material property variations as well as load estimation from many other disciplines such as 
Thermal and fluid. Thus in order for a design system to be used on practical product design it 
must be capable of accommodating multi-disciplinary analysis. The multi-disciplinary 
definition should encompass traditional engineering models such as structures, thermal fluid 
etc. as well as models representing non-engineering functions such as cost models. 


Fundamentals that Drove the RDCS Development 

Multi-Disciplinary Approach a Must 
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RELIABLE PRODUCE DESIGNS CAN BE ACHIEVED 
USING A VARIETY OF APPROACHES 


The RDCS system enables robust designs in a variety of ways. Fundamentally, it 
allows the engineer to understand the design space in a systematic manner both in the 
deterministic as well as in the non-deterministic sense using a variety of tools. Efficiency in 
the problem statement, computations and user friendliness is of paramount importance. 
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RDCS TOOL - AN INSTANCE OF MODERN DESIGN FRAMEWORK 


The RDCS architecture cleanly separates the key elements of the design task to meet 
the objectives discussed earlier. The domain specific computational models are defined in the 
mathematical models (can be a network of multi-disciplinary models). The design processes 
contain views and associated algorithms that enable the engineer to understand the design 
problem in a variety of ways. They allow the engineer to cut and slice the design space to 
under stand the design problem efficiently. It is important to recognize that theses design 
process views and algorithms are completely generic and have no math model domain 
intelligence in them. The system director provides the communication interface between the 
three elements. 
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DESIGN SPACE IS SAME FOR ALL THE DESIGN PROCESSES 


In the past, sensitivity analysis, parametric analysis, design of experiments using full 
and fractional factorial designs, probabilistic analysis either Monte Carlo simulation or First 
or Second Order Reliability Methods, Optimization, Taguchi Methods etc. were viewed as 
independent approaches to product design using separate computer programs. This slide 
highlights that all the design approaches are trying to solve the same design problem but with 
a different sampling approach of the design space. Of course the analysis of information have 
different approaches. 

A single design framework that integrates and implements all approaches has 
significant advantages. This integrated approach provides opportunities for avoiding wasteful 
duplication in input preparation work, allows exploitation of synergy between approaches 
using common compute modules, sharing results between deign processes to avoid redundant 
or duplicate computations. The results are improved overall efficiency in computations as 
well as quantum leap in design knowledge gained. 
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AVAILABLE OPTIONS AND TOOLS WITHIN RDCS 


The RDCS system is one instance of such a modem design framework with probably 
many more frameworks to come in the future. These frameworks when implemented with a 
clean architecture that separates the tasks (as earlier discussed), addition of new yet unknown 
design processes in to the frame work should be an easy matter. In order for these frame 
works to be used in the design shops, easy graphic user interfaces to input as well as to post 
process the results is a must. They should further encapsulate the complexities of the analysis 
and the technologies they represent. 


Available Options and 
Tools Within RDCS 



Robust Design ComputaUori^aLSy st^: example I 


FBe System Monitor 
Run Status 
Slop Project 

/ c Component 


todel Design 
w®T- Process 

Selected Design Condition 
Sensitivity Analysis 
F ypical/Fxtmno Designs 
Design Space Scan 
Deterministic Optimization 
Taguchl Metliod 
Probabilistic Analysis '-"C, 
Design Scan -- 


Launch 

Setup C 


: Selected-!^ 
Sensitivity/ 
1 Design Spa 
I Determinfsi 
J Taguchi Me 
Probabillsd 


l/Extreme Designs 


Response Surface ^ 

:l: — ™ Create/Retrieve Set] 
Visualize j 


Monte Carlo 

Approximate Analysis j 


Analysis Methods] 


Optimization i 


«• f ju-wil ... ! M MJ U U u . .j,,,,,,,. 

[ New Proba bilistic Methods 




Rocketdyne 
Propulsion & Power 


12 




Figure 12 


238 






WHY DO WE NEED DISTRIBUTED COMPUTING? 


The systematic exploration of the design space with many different design process 
views comes with a price, increased computational load. The number of cases or analysis that 
need to performed can be one to several orders of magnitude greater than single one worst 
case analysis. However, we are in the midst of a computer revolution wherein the cost of 
computing has gone down significantly. Software that implements a robust distributed 
computing technology wherein hundreds of cases can be run in parallel can make a 
significant difference in the use of these technologies in a schedule driven product design 
shop. 


Why Do We Need Distributed Computing? 


• Understanding of the design space comes at increased 
computational cost 

• Sensitivity analysis - finite difference(2n) 

• Design scan - (partial, factorial, DOE) ( n m ) 

• Response surface ( n sampling points) 

• Optimization (iterative) 

• Taguchi analysis (orthogonal array) 

• Probabilistic analysis (Monte Carlo simulation or other) 

• Probabilistic sensitivity analysis (variability sensitivity) 

• Probabilistic optimization 

• All involve one to three orders of magnitude more function 
evaluations than the traditional approach 

• A design framework with out a robust distributed computing 
element is not a scaleable practical tool for design shops 
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DISTRIBUTED COMPUTING AND INTERNET TECHNOLOGY 


The internet revolution is developing technologies that will make it easier to 
implement the distributed computing model in engineering design. It is prudent to develop 
engineering design framework software solutions that leverage the enormous investments 
and progress made in the computing industry to assemble virtual design teams and 
collaborative solutions. 


Distributed Computing And Internet Technologies 


• Distributed Computing is the basic element of internet 

service model 


Client 2 


• Collaborative computing client V 

• virtual design teams 

• Geographically separated 

• Web enabled design environment 

• Modern tools makes it easier to implement a distribute 
computing model 

• CORBA, JAVA/RMI/IIOP 

• Web Browser, Applets, Servlets c , 


Service 


Internet 

Network 

Connection 


• Internet technologies provide the glue to integrate 
• Heterogeneous computing platforms 


UNIX, PC/NT, LINUX 
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GRAPHICAL MATH MODEL CONSTRUCTION AND DISPLAY 


The following slides are actual screen shots from the RDCS system. The left palette 
represents suite of analysis tools that a typical design shop might use in their design process. 
The palette is site configurable and the suite of tools can change from one design shop to 
another. 

The user simple clicks the icons in the palette and drops the analysis code in the 
sketch pad repeatedly to construct a network of codes to be executed in a particular order 
determined by the arrows. This network of functional models with arrows representing the 
information flow represents the actual design process used for that component perhaps 
representing many domains. The live buttons and arrows in the sketch pad can be clicked to 
provide additional screens completing all the information that is needed to execute the entire 
sequence. 

It is very common to achieve several orders of magnitude improvement in the design 
cycle time using this automate feature of the RDCS system over conventional manual 
execution of codes in sequence. 



Graphical Math Model Construction and 

Display 


— 

* Bottoms Up or Top Down Construction 
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DETERMINISTIC DESIGN CONCEPTS 


The RDCS philosophy is to provide the engineer with information that he is used to 
as well as provide the more extensive design space exploration results. In this case, the 
conventional deterministic worst-case analysis provides information that can be used to 
arrive at safe designs but does not provide detailed information about product performance at 
other design conditions or at an another point in the design window. 


Figure 16 
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Deterministic Design Concepts 


• Deterministic typical worst case design provides information about margins but 
frequently more information is needed 









UNDERSTANDING THE DESIGN SPACE - DESIGN SENSITIVITY PROVIDES 

VARIABLES RANKING 


The deterministic sensitivity analysis provides a way to rank the input variables based 
on their effect on output performance. This can help the engineer to determine which knob to 
turn to achieve the desired effect. Typically this is done at a nominal design point. If the 
behavior is nonlinear it is necessary to perform several sensitivity analysis at different design 
points. 



Understanding the Design Space 

Design Sensitivity Provides Variables Ranking 


Bar Chart 


cost-beta — Sensitivity Analysis 


Plot Type: Tornado Plot 
■ Sorted 


cost_beta — Sensitivity Analysis 
Design Instance — sens_hi__low 




Insurance 




Refresh \ 
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UNDERSTANDING THE DESIGN SPACE 
- DESIGN SCAN VIEW OF THE DESIGN SPACE 


A detailed design space scan of the effect of input on output over the entire deign 
window is one of the most powerful feature of the design space exploration. It provides the 
engineer a comprehensive view of the product behavior over the entire range (from benign 
linear to highly nonlinear). The use several types of factorial designs and the available 
computing power to obtain the desired information makes this a practical and often used 
feature of the RDCS system. 
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UNDERSTANDING THE DESIGN SPACE - RESPONSE SURFACE 


The response surface generation is a very useful feature of the RDCS system not only 
to visualize but also to use the response surface as a surrogate model in probabilistic or 
optimization design processes. 





DAMAGE 


Figure 19 


Understanding the Design Space 

Response Surface 
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UNDERSTANDING THE DESIGN SPACE - DETERMINISTIC OPTIMIZATION 


The mathematical optimization design process is one of the many tools available in 
the RDCS system. The key point here is for the engineers to be sensitive to the presence of 
variations of the design variables and make an assessment of the effect of variations on the 
objective function. 



Objective Function 


Understanding the Design Space 

Deterministic Optimization 


• Minimizing an objective 
function subject to 
constraints 


Display Entity: tralnt Function- Hi 
Plot Type: ZDCurveLabeipiot 
X-Axis Label 


OptimaAOS: 3_bar_optima 
Instance Name: optl 
Constraint Function History 


Display Entity: Generic X-Y-Z Plot 
Plot Type: 3DSurfacePlot 

X-Axis label 


OptimaAOS: 3_bar„optima 
Instance Name: optl 
Generic X-Y-Z Plot 
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UNDERSTANDING THE DESIGN SPACE - TAGUCHI VIEW OF DESIGN 


Taguchi analysis is very popular at the production shop in which the experiments are 
performed using the actual hardware. In the context of the RDCS system, the experiments are 
performed on the numerical models, but otherwise the Taguchi process remains the same. 

The automatic selection of the orthogonal array based on built-in features inside the 
RDCS system makes it practical to perform this type of analysis at a product design shop. 
The design cycle time improvement over the manual methods can be several orders of 
magnitude. 



Figure 21 
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UNDERSTANDING THE DESIGN SPACE - PROBABILISTIC ANALYSIS 


Use of the probabilistic analysis will provide quantified reliability estimates. There a 
variety of distribution models such as Normal, Weibulll, Exponential etc., are available to 
model the input variations. The output variations can be represented as Probability Density 
Function, Cumulative Distribution Function as well Frequency Diagrams, Scatter Plots etc.. 
The key point here is that when probabilistic results are viewed along with extensive design 
space exploration and other design process views of the product, the engineer gains a 
comprehensive knowledge of all aspects of expected product behavior. 





Understanding the Design Space 

Probabilistic Analysis 


• Provides the effect of 
variability and/or 
uncertainties on design 
performance 
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RDCS APPLICATIONS 


The RDCS system has been successfully used in a variety of industries such as 
automobile industry, space industry as well in airplane industry. The breadth of the 
applications lends credence to the design framework concept with all the benefits discussed 
earlier and is in fact filling in a customer demand. 


RDCS Applications 




Automotive 


Air Frame 


Rocket Propulsion 
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CONCLUDING REMARKS 


The challenge is to design and build products meeting cost and schedule constraints 
effectively. The product should perform as expected the first time. Early indications are that 
the use of design frame works such as the RDCS system have great potential to meet that 
need. 
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Overview of Probabilistic Risk Assessment at NASA: 
Past, Present and Future 


Dr. Michael G. Stamatelatos 
Office of Safety and Mission Assurance 
NASA Headquarter, Code Q 
Washington, D.C. 20024 


251 



RARE THINGS DO HAPPEN 


PRA adds the “probability” dimension to the traditional “deterministic” dimension of 
engineering and safety analyses. Although PRA, as a discipline, has been the evolutionary 
product of only the last couple of decades, much of the underlying methodology has been in 
use for a long time. In fact, the concept of probability for describing the likelihood that rare 
events happen is ancient and dates from Ancient Greece. 



Figure 1 
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NASA MANAGES RISK ON A DAILY BASIS 


NASA is an organization that deals with unique and pioneering technological projects 
essentially routinely. Therefore, NASA deals with the concept of risk, either explicitly or 
implicitly, on a daily basis. Also, safety is a top priority at NASA and it is being applied to 
public, astronauts and pilots, personnel, and property in exactly this order. 




issiort Su 


:s Starts With Safety 


NASA Manages Risk on a Daily Basis 


• As a technological pioneer, NASA 
has, explicitly or implicitly, evaluated, 
accepted and managed risks 
throughout its existence 

• NASA places 

(1) public safety, 

(2) astronaut and pilot safety, 

(3) personnel safety, and 

(4) property safety 

at the very top of its priorities 


Figure 2 
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NASA PRA VISION 


Our vision is for NASA to achieve world-class state-of-the-art capability in 
performing and using PRA in technical and management decisions. Since NASA is a leader 
in many technical fields in which it is involved, we hope, before long, to see NASA become 
a leader in this field also. 


■- ' ■ ' : I’ 1 ■ ■ ! . . . 

n/lission Success Starts With Safety 

Vision 


Develop and maintain a world-class in- 
house capability to perform, manage, and 
use Probabilistic Risk Assessment (PRA) 
methods for the benefit NASA personnel 
and programs 



Figure 3 



NASA PRA OBJECTIVE 


NASA’s objective is to develop and use state-of-the-art PRA methodology to help 
ensure mission success, improve safety throughout product life cycle, improve performance 
and, eventually, reduce design, operation, and maintenance costs. 



NASA’s PRA objective is to use state-of-the-art PRA 
methodology to support management decisions to: 

• Ensure mission success, 

• Improve safety in design, operation, 
maintenance and upgrade, 

• Improve performance, and 

• Reduce design, operation and 
maintenance costs 


Figure 4 
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IT WAS NOT ALWAYS THAT WAY... 


Some of the important techniques currently used in a PRA (e.g., fault trees) were, in 
fact, developed some forty years ago in connection with aerospace applications. Early in the 
Apollo project, a pessimistic probabilistic estimate of mission success, calculated to be 0.2, 
strongly disappointed NASA managers. Nevertheless, NASA was not deterred by this 
prediction and, later on, experience showed a much higher probability of success. Thus, the 
credibility of PRA was lost for many years at NASA who proceeded with the use of only 
Failure Modes and Effects Analyses (FMEA) in support of their safety assessments. It was 
not until the Challenger accident and mainly after recommendations from outside experts that 
NASA started again to perform PR As. 
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It Was Not Always That Way ... 


• Early Apollo program estimate of mission success probability was 
a disappointing 0.20. 

• However, between 1 969 and 1 972, 6 out 7 successful Apollo 
missions demonstrated 0.86 mission success probability. 

• This discrepancy caused dissatisfaction with PRA at NASA and 
reliance on FMEAs. 

• October 29, 1986 - The “Investigation of the Challenger Accident” 
by the Committee on Science and Technology of the House of 
Representatives criticized NASA for not “estimating the 

probability of failure of the various [Shuttle] elements. ” 

• January 1988 - In the “Post-Challenger Evaluation of Space 
Shuttle Risk Assessment and Management,” the Slay Committee 
recommended that “probabilistic risk assessment approaches be 
applied to the Shuttle risk management program at the earliest 
possible date." 


Figure 5 
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PRA RENAISSANCE AT NASA 


Between 1987 and 1995 consultants by NASA performed some fifteen probabilistic 
studies. Also, the Administrator encouraged the initiation and development of an integrated 
computerized PRA tool, QRAS. In spite of these initiatives, the use and credibility of PRA at 
NASA did not grow significantly because some important ingredients were still missing. 


E ■ Mission Success Starts With Safety 

PRA Renaissance at NASA 

• Between 1987 and 1995, some fifteen PRA studies were 
performed for NASA 

• In July 1996, NASA Administrator Dan Goldin requested “a tool 
to help base (Shuttle) upgrade decisions on risk.” 

• In October 1997, an early version of the NASA Quantitative Risk 
Assessment System (QRAS) was demonstrated to the 
Administrator. 

• In February 1998, Version 1.0 of QRAS was baselined. 

Unfortunately, the PRA efforts during this PRA revival era have 
found little understanding and usefulness at NASA because 

important basic ingredients were missing 


Figure 6 
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ACQUIRING PROVEN INGREDIENTS FOR SUCCESS 


Applications of PRA in other industries, notably nuclear, have shown that optimal 
understanding, appreciation, and use of PRA techniques are not realized by an organization 
until in-house personnel of this organization gain important achievements: sufficient in-house 
expertise to manage and use PRA for management decisions; in-house ownership and 
corporate memory of PRA methods, computer tools, data, and results; and transfer of PRA 
technology to the in-house decision makers who use PRA and its results. NASA is 
aggressively pursuing approaches to reach these achievements. 
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Acquiring Proven Ingredients for Success 


♦ In-house expertise to perform, manage and use 
PRAs to make sound decisions 

♦ in-house ownership and corporate memory of 

PRA methods, tools, databases and results 

♦ Transfer of PRA technology to in-house personnel 
and managers who are the ones who need to 
manage, oversee, understand, and use PRA to 
make sound management decisions 


Figure 7 
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RELATIONSHIP BETWEEN RISK MANAGEMENT AND PRA 


Risk of interest at NASA is both of a technical nature and of a programmatic nature, 
the latter being related to program costs and schedule. Risk assessment covers of the first two 
elements of risk management: risk identification and risk analysis. Risk assessment can be 
performed either qualitatively or quantitatively. Techniques like failure modes and effects 
analysis (FMEA), fault tree analysis (FTA), master logic diagrams (MLD), event sequence 
diagrams (ESD), and event tree analysis (ETA) can be used in both qualitative and 
quantitative risk assessments. Additionally, statistical and actuarial techniques, as well as 
simulation techniques, can be used in quantitative risk assessments. Risk assessment results, 
in conjunction with decision analysis techniques, are used to formulate risk prevention and 
mitigation plans. The last two elements of risk management, risk tracking and risk control 
must be consistent with the organization management system. 



, Mission Success Starts With Safety 

Relationship Between Risk Management 
and Probabilistic Risk Assessment (PRA) 
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Technique Application 
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ONE- YEAR ACCOMPLISHMENTS 


Within the past year, NASA has made great progress toward reaching its PRA 
objectives: (1) a policy for PRA use was drafted and is being circulated for comments; (2) a 
practitioner’s PRA procedures guide for aerospace applications was drafted; (3) PRA training 
at NASA was conducted several times at both awareness level and practitioner’s level; (4) 
more than 90 NASA people were trained on SAPHIRE, chosen by NASA as its “baseline” 
integrated PRA program which, together with NASA’s own program, QRAS, will be used 
for PRA training and PRA projects; (5) PRA information exchanges have been organized in 
the form of workshops where NASA personnel can share PRA information and experience; 
(6) NASA is cooperating in the field of PRA with US organizations that are experienced in 
PRA, like the Nuclear Regulatory Commission (NRC), and with foreign aerospace agencies, 
e.g., the European Space Agency (ESA) and the Japanese space agency, NASDA. 
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One-Year Accomplishments 
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PRA POLICY REQUIREMENTS 


NASA is performing PRA for a wide range of programs/projects ranging in nature 
and detail from conceptual to detailed design, operation and/or system upgrade. Some 
projects involve human participation while others do not. When the public may be affected, 
NASA and federal requirements make it necessary to conduct a full-scope PRA. NASA is 
also performing full-scope PRAs for all human space flights. For non-human rated space 
projects, the PRA requirements are somewhat different. For programs of high strategic 
importance, high schedule criticality, or high cost a full-scope, PRA is also required, 
sometimes at a reduced scope if appropriate. Although PRA is recommended for all projects, 
only a limited level PRA may be required for lower cost programs and perhaps none for very 
low cost projects that are not human rated. 
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PRA Policy Requirements 


CONSEQUENCE CRITERIA / SPECIFICS 

CATEGORY 

Human Safety & Public Planetary Protection 

Health Safety Program Requirement 

White House Approval 

|(PD/NSC-25) 

Human Space Flight 


Mission Success High Strategic Importance 

(for non-human rated High Schedule Criticality 
missions) __________________ 

Higher-Cost Missions (>$100M) 


|Lower-Cost Missions (<$100M) 


(*) LEGEND: 


F = Full Scope; L = Limited Scope; N= 


NASA PROGRAM/PROJECT 
_ (Classes and/or Examples) 

Mars Sample Return 

Nuclear payload (e.g., Cassini, 

_ Ulysses, Galileo) 

International Space Station 

Space Shuttle 

Crew Return Vehicle 
_ Mars Program 
Launch window (e.g., planetary 
missions) 

Earth Science Missions (e.g., EOS) 
Space Science Missions (e.g., SIM) 
Technology Demonstration and 
_ Validation (e.g., EO-1) 

Earth Science Missions (e.g., 

QUICKSCAT) 

Space Science Missions (e.g., 

HESSI) 

Technology Demonstration and 
_ Validation (e.g.. Deep Space 1) 

None 


Il’RA 

SCOPE* 
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QUANTITATIVE RISK ASSESSMENT SYSTEM (QRAS) 


QRAS is a NASA integrated PRA computer program that is in many ways similar to 
SAPHIRE and in some ways different. Like SAPHIRE, QRAS uses event trees and fault 
trees to model and quantify accident scenarios. Like SAPHIRE, QRAS has the capability to 
perform dependent failure (also known as “common-cause failure”) analysis and uncertainty 
analysis. Unlike SAPHIRE, QRAS leads the analyst through the development of a hierarchy 
for hardware systems, sub-systems and components that facilitates identification of hardware 
failure modes as accident sequence initiators. Unlike SAPHIRE, the scenarios in QRAS are 
fist developed into event sequence diagrams (ESD), which tend to be easier to understand 
and develop by system engineers who are not experts in PRA. These ESDs are then 
transformed into equivalent event trees by the software, without analyst intervention, before 
the sequence is quantified. Version 1.6 of QRAS was recently released and is now in beta 
testing. Thus, NASA is in the enviable position of having two state-of-the-art integrated PRA 
computer programs instead of one. 
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TYPES OF RISK AND RELATED CONSEQUENCES 


Risk assessments may be performed for a variety of reasons and associated 
consequences. These include: safety, environmental impact, cost risk, programmatic risk, 
mission success, etc. Risk assessments for these different types of applications are generally 
different in scope and may require the use of different types of techniques for modeling and 
quantification. 
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Types of Risk and Related Consequences 


SAFETY 


ENVIRONMENTAL 


I Deaths, injuries, illness 


COST 


Contamination, loss of use 
Money lost 


PROGRAMMATIC 




Mission, schedule, 
etc. 


OTHER ? 

COMBINATIONS ? 


Figure 12 
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PRA SIMPLY DESCRIBED 


A PRA generally provides the answers to three types of questions: (1) what can go 
wrong? (i.e., what are the scenario initiators?) (2) how frequently does this happen? (i.e., 
what are the scenario probabilities or frequencies?) and (3) what are the consequences if 
something goes wrong? (i.e., what is the scenario consequences severity?) The final product 
is a description and interpretation of the risks being assessed including their numerical values 
and uncertainties if the assessment is quantitative. 
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PRA Simply Described 
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2. HOW FREQUENTLY DOES IT HAPPEN ? 
(SCENARIO FREQUENCY QUANTIFICATION) 


RISK STATEMENT 


3. WHAT ARE THE CONSEQUENCES ? 
(SCENARIO CONSEQUENCE QUANTIFICATION) 



CONSEQUENCE 

MODELING 
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EXPECTATIONS FROM A PRA 


The expectations from a PRA are generally assistance in: preventing or mitigating 
mishaps, mission success or performance enhancement, safety improvements and/or design, 
operation, and maintenance cost reductions. 


K1ISE * Mission Success Starts With Safety 

Expectations from a PRA 

• Mishap prevention and mitigation 

• Mission/performance success enhancement 

• Safety improvements throughout life cycle 

• Design/operation cost reduction 



Figure 14 




SCENARIO-BASED PRA MODELING APPROACH 


The scenario-based PRA modeling approach for safety applications can be 
diagrammatically shown to consist of: technical or programmatic information collection and 
analysis; identification of initiators with techniques like the master logic diagram; event 
sequence development; system failure assessment using fault trees; and evaluation and 
analysis of the results. 
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Scenario-Based PRA Modeling Approach 


Figure 15 
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ELEMENTS OF SCENARIO-BASED PRA 


The quantitative scenario-based PRA starts with objective definition and proceeds 
through the following general steps: system familiarization; initiating events identification 
and selection; scenario, or event-sequence modeling (e.g., using ESD or ETA); failure 
modeling (e.g., using FTA), quantification and integration of the scenario risks; uncertainty 
analysis; and interpretation of results. Scattered throughout the PRA process is data 
collection and processing as necessary. Depending of the analysis requirements, sensitivity 
analyses and/or importance ranking analyses are also performed. 





nr nr nr nr -O. 



Full application of a scenario-based PRA 
involves a number of steps shown here 
diagrammatically to illustrate the process 
that starts with the definition of objectives 
and ends with the interpretation of results 



Figure 16 
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RISK SOURCES IN SAFETY RISK ASSESSMENT 


In a safety risk assessment, the risk contributors can be either, all or combinations of 
the following types of sources: hardware failures, “external” events or acts of nature, human 
error, or errors due to organizational structures or practices which are globally called 
“organizational factors.” These risk sources are listed here in order of increasing modeling 
complexity and of decreasing maturity and accuracy in the methodology, modeling and 
associated data quantity and quality that are generally available to perform risk assessments. 
Not listed in this diagram, but also very important for NASA applications, are software 
failures that are currently not being modeled very well because of general lack of data and of 
sophistication in the analysis methods. 
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MAJOR NASA PROGRAMS NOW USE PRA 


The Space Shuttle and the International Space Station are two major human rated 
NASA projects for which PRAs are being currently performed. The first one is primarily 
performed to assist the space shuttle upgrade program. The second one is performed to assist 
construction, assembly and operation of the international space station, one of the most 
challenging scientific and engineering challenges of our time. PRA has been used to assign 
safety goals to the space shuttle program. 
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Major NASA Programs Now Use PRA 


Space Shuttle Development Roadmap 



Goals and 1 
Objectives 

97 

02 

07 

12 

1 Fly Safely 

; 1 vehicle 
: loss in 148 
flights j 

t vehicle loss 
in 250 flights 

1 vehicle loss 
in 325 flights 

1 vehicle loss 
in 500 flights 


International Space Station PRA 


• 1999 -- The NASA Advisory Council 
recommended, the NASA Administrator 
concurred, and the ISS Program has 
begun a PRA. 

- The modeling will be QRAS- 
compatible. 

- First portion of PRA (through Flight 
7A) - delivered in Dec. 2000. 
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MARS SAMPLE RETURN MISSION 


Because it is unknown what types of health or other risks a sample from Mars could 
pose upon return to Earth, NASA’s Planetary Protection Program has preliminarily assigned 
the probability of one in a million or less that such a sample will contaminate the Earth. This 
means that the sample should have less than one in a million chance of being opened in an 
uncontrolled environment upon or after re-entry in the Earth atmosphere. This type of goal 
cannot be satisfied by any assessment other than a probabilistic risk assessment (e.g., not by 
any deterministic safety assessment). 
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liars Sample Return Mission 



♦ Mission must meet a Planetary Protection Program (PPP) 
criterion of <1 O' 6 probability of Earth contamination upon 
return of sample 

♦ PRA is used to evaluate mission compliance with the PPP 
criterion 
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NASA’S UNIQUE PRA METHODOLOGY NEEDS 


Although NASA is acquiring or adapting PRA methods, software and data from 
applications in other industries to the extent possible, NASA PRA applications have some 
unique features that require special needs and special attention. They include: diversity of 
project application; multi-phase treatment; unique types of initiators; unique environments; 
unique types of adverse consequences; special treatment of human and software reliability; 
etc. These capabilities need to be developed or acquired as soon as possible. 
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NASA’s Unique PRA Methodology Meeds 


• Broad range of programs: Conceptual non-human rated science 
projects; Multi-stage design and construction of the International 
Space Station; Upgrades of the Space Shuttle 

• Risk initiators that vary drastically with type of program 

• Unique design and operating environments (e.g., microgravity 
effects on equipment and humans) 

• Multi-phasing approach in some scenario developments 

• Unique external events (e.g., micro-meteoroids and orbital debris) 

• Unique types of adverse consequences (e.g., fatigue and illness in 
space) 

• Different considerations for human reliability (e.g., astronauts vs. 
other operating personnel) 

• Greater importance of software reliability 

• Specialized database needs 


Figure 20 
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END STATES FROM ISS PRA 


Examples of unique PRA end-states can be seen in the International Space Station 
(ISS) PRA: loss of station; loss of crew; evacuation end states, and other undesired end states 
like loss of module; loss of system; or collision between a US or a Russian logistic vehicle 
and the ISS. 
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End States from ISS PRA 


•Station and Crew are Functional 
(OK) 

• This end state signifies that the 
station is still working with the 
flight rule constraints 

•Critical End States 

-Loss of Station and Crew (LOS) 

- Catastrophic loss of the station and crew 

-Loss of Crew (LOG) 

• Resultant loss of a crew-member 

• Also includes the inability to evacuate the 
station due to evacuation end state and the 
unavailability of either Soyuz or Orbiter to 
perform such a task 

-Evacuation End States (EVAC) 

• Emergency Evacuation 

• Flight Rule Evacuation 

• Medical Evacuation 


•Other Undesired End States 

(OUE) 

-Loss of Module (LOM) 

• The shut down of any pressurized 
module as dictated by flight rule or as 
result of MMOO 

-Loss of System (LOSys) 

• The loss of either US or RS distributed 
systems 

• Loss of a function such as 

- ability for Orbiter, Progress, or Soyuz 
to dock 

- ability to reboost 
insufficient 0 2 or N 2 reserves 

-Collision (COL) 

• impact of the Orbiter, Progress, or Soyuz 
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INITIATING MEDICAL EVENTS FROM IS PRA 


The ISS PRA also demonstrates the need for special types of initiating events that are 
normally not dealt with in ground based systems or facilities. Medical initiating events due to 
illness or adverse environment in space are examples of unique types of initiators that may be 
encountered in NASA PRA applications. 
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initiating Medical Events from ISS PRA 


• Medical disorders quantified are for “severe” injuries or 
illnesses only 

- those that would normally require hospitalization on Earth 

• Medical categories are: 

- Circulatory 

- Dermatology 

- Digestive 

- General Internal Medicine 

- Genitourinary 

- Infectious Disease 

- Neurology and Psychology 

- Respiratory 

- Trauma and Poisonings 


Figure 22 
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ENERGETIC HAZARD ESD EXAMPLE (ISS PRA) 


Another example of a unique type of initiator in a NASA PRA is impact with micro- 
meteoroids or orbital debris. Such impact with ISS systems and components can lead to very 
severe consequences to the space station and its personnel. Equally severe consequences can 
result from this type of collisions with ISS personnel performing extra-vehicular activities. 
NASA has a number of computational models and tools for conducting probabilistic 
assessment of these types of collisions and associated risks to hardware and ultimately to ISS 
personnel. 
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Energetic Hazard ESD Example (ISS PRA) 


Micrometeoroid and Orbital Debris 



PNCF 


Figure 23 
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GOALS FOR THE FUTURE 


Our vision for future PRA use at NASA shows not only a world-class capability to 
perform PRA for all applications for which this method is appropriate but also the creation at 
NASA of the right culture and environment to make the best use of PRA in all or most 
management decisions that impact mission success, performance improvement, safety 
enhancement and costs. 




Mission Success Starts With Safety 


Goals for the Future 


• Risk awareness enhancement 

• PRA/QRA training of project managers, 
astronauts and operational personnel 

• Agency-wide risk informed culture 

• PRA to become a way of life for safety and 
technical performance improvement and 
cost reduction 

• PRA for readiness review support 

• Risk-informed management process 


Figure 24 
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“LOSSES TO SOCIETY AND OPPORTUNITIES FOR COMPANIES” 


At the heart of “non-de termini Stic” methods is the inclusion of variation in 
engineering calculations. Unlike engineering calculation examples in textbooks, 

demonstrating exact dimensions, material properties, and boundary conditions, these 
calculations are effectively made inaccurate by variation. Given variation in input 
conditions, answers resulting from the associated engineering calculations will also exhibit 
variation. The certainty desired from deterministic methods implodes because it excludes 
variation and is replaced by the uncertainty of non-deterministic methods. Embedded in this 
presentation is an introduction to efforts underway Boeing to foster a greater degree of 
awareness of variation in everyday activities. The title of this presentation is a reference to 
the opportunities that exist to “reduce loss to society” when companies develop a deeper 
appreciation of variation, including its sources (connections and systems) and its 
consequences. 
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AGENDA 


This overview presentation will include a statement of the objectives, a review of 
assumptions, a simple systems thinking exercise involving woodworking, an introduction to 
investment thinking, and a vision of the potential of “better thinking” about variation, 
systems, psychology, and the theory of knowledge. 



a Objective 
t. Assumptions 

■ Cutting Wood 

■ Investment Thinking 

* The Role of Better Thinking 
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OBJECTIVE 


The material presented in this overview is an accumulation of 15 years of reflection 
on management theories contributed by Charles Kepner, Benjamin Tregoe, W. Edwards 
Deming, Genichi Taguchi, and Peter Senge, among many others. My first exposure to these 
“change leaders” was the problem solving and decision making analyses of Drs. Kepner and 
Tregoe, to be followed soon after by the work of Dr. Taguchi and Dr. Deming. In repeated 
applications involving highly visible problem resolution activities, the potential energy of 
Taguchi Methods became more and more evident. The early applications also revealed the 
narrow focus of these applications - to fix or repair products and processes. An obvious 
application pattern was developing. In borrowing from the concepts of Dr. Deming, a theory 
was developed to explain why this costly application pattern was widespread across 
industries. Dr. Deming’ s management theory provides an explanation for how organizations 
can develop higher levels of working together. The objective of this presentation is to offer a 
view of the potential energy offered by the synergetic linkage of these varied management 
theories. 



Introduce the potential energy of 
integrating the management theories 

of 

Dr. W. Edwards Deming 

S I Dr. Genichi Taguchi 

1 and others... 

Bill Bellows (7i 0£7JEf/V£w 3 


Figure 3 



281 



3-D DIFFUSION EQUATION 


My engineering background has its roots in heat transfer analyses, solving this partial 
differential equation to establish the temperature distribution of an object under a variety of 
boundary and initial conditions. The terms of the equation include a “diffusion” component 
on the left side to represent the effects of three-dimensional heat conduction. The “Q” 
component is the so-called energy “source term”, included as an indication of the magnitude 
of “energy generation” within the object. The right side of the equation is the sole time- 
dependent term; an indication of temperature changes over time. As will be shown in the 
closing chart, my awareness of these three terms has provided general guidance in the 
diffusion of better management theories. 




of Heat 


3 2 T/ 3 x 2 + 3 2 T/ 3 y 2 + 9 

2 T/ 3 z 2 

+ Q’”/k = 

l/a 3 T/ 3 1 
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QUALITY LOSS 


Dr. Taguchi has uniquely defined quality in terms of "minimizing loss to society". 
This presentation will offer a view of ongoing activities at The Boeing Company to promote 
better thinking as a means to achieve better doing, which can be translated into minimizing 
losses to society and Boeing. These efforts involve a focus on variation management, seeing 
systems, and investment thinking. 



"Quality is the (minimum of) loss a 
product causes to society after 
being shipped, other than losses 
caused by its intrinsic functions. " 

Dr. Genichi Taguchi 

Source: Introduction to Quality Engineering, Dr. Genichi Taguchi 
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PERCEPTION & THINKING 


Tom Johnson, a professor of Quality Management at Portland State University, offers 
as a ponderable that “what we see” is a reflection of “how we think”. As a simple example, 
consider the response of a parent upon reading their child’s report card. Are the grades, low 
or high, an indication of the student, alone? Or, might it be possible that the grades are an 
indication of the “education system” in which the child resides, a system that includes the 
student, the teacher, the parents, and the hiring practices of the local board of education, to 
name a few inter-dependent parts. How the parent responds to their child’s report card will 
reveal their understanding of the size of the child’s education system and the degree of 
interconnectedness of the components. Likewise, in a factory setting, the quality of a duct 
weld is a reflection of the welding system in which the welder resides. 



"How the world we perceive 
works depends on how we think. 

The world we perceive is a world 
we bring forth through our thinking. " 

H. Thomas Johnson 
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UTILIZATION OF THINKING 


In the spirit of thinking about interconnections and systems, consider the logic of 
routinely asking questions such as “Where are we going?” Where does this fit in?” “Where 
did this come from?” “What is my role?” and “What is this part of?” To ponder them 
routinely is to be reminded of a general pattern of inter-dependence that surrounds all of us. 
The last question, “Where should we invest?” explores the implications of systems thinking, 
when coupled to economics. The concept of connections is fundamental to recognizing the 
relationships between upstream, local, and downstream conditions. 



■ Where are we going ? 

* Where does this fit in ? 

* Where did this come from ? 

■ What is my role ? 

* What is this part of ? 

* Where should we invest ? 
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ASSUMPTIONS 


From my reflection on a 15-year review of management theories, I would like to 
propose three progressive assumptions; 1- A better way to operate an organization is to invest 
resources with the ability to delight customers, 2- Better investment results from discovering 
opportunities to invest, and 3-The discovery of opportunities for investment is limited by 
how thinking is conditioned. 



■ A better way to operate an 
organization is to invest resources 
with the ability to delight customers 

a Better investment results from 
discovering opportunities to invest 

■ The discovery of opportunities for 
investment is limited by how thinking 
is conditioned 
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Figure 8 
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WHAT IS NEEDED? 


My conclusion is that we are in need of better thinking, that is - “thinking that 
promotes better discovery.” Discovery of opportunities, discovery of connections, discovery 
of systemic effects, to name a few forms of needed discovery. If one cannot see connections, 
then one can see only parts. As such, our lives and the world we live in are viewed as sets of 
fragmented pieces. Without a sense of connections, we are resigned to being reactive. We 
tend not to “see things coming” and, consequently, experience problems without warning. 
Adding to this scenario of disconnections, we then act to impart blame to elements of the 
system instead of to the system itself. Such blame may be imposed on the student in a 
classroom or on the welder on a shop floor. If only we could see connections, we could 
anticipate. Such anticipation provides early warning of impending trouble and the ability to 
pro-act. Better thinking offers the ability to uncover these opportunities for investment. 



What is needed ? 


Bill Bellows 




Figure 9 
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QUESTION 


Consider the simple question, “What is this part of?” Imbedded in this question is an 
explicit reference to a connection. The systemic thought is revealed by the concept “part of’, 
as opposed to “part”. Without the “of’, we could only inquire about the part, as in the 
question, “What is this part?” Given this inquiry, the connections would be lost as we return 
to a worldview of “fragmented pieces”. 


Question 


m What is this a part of ? 
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Figure 10 
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WATER AND ROCK LOGIC 


In reference to the “from-this-to” sequence, questions such as “What is this part of? 
Where did this come from?” and “What will this lead to?” represent the essence of 
understanding relationships and inter-connections. The thinking revealed by these questions 
has been termed “water logic” by the noted “thinker” and author. Dr. Edward de Bono. By 
contrast, references to events, parts, and pieces, are termed “rock logic”. To view the world 
with “rock logic” is to view it in the form of an “exploded view” - parts without connections. 
To view the world with “water logic” is to view without seeing parts. Such a view reveals 
the world to be a pattern of relationships and linkages. 


Water 

Logic 



What is this a part of ? 
Where did this come from ? 
What will this lead to ? 


Rock 

Logic 


■ Event focussed 

■ Part focussed 

■ Piece focussed 



Source: Water Logic, Dr. Edward de Bono 
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QUESTION: CUTTING WOOD 


To better appreciate the implications of water and rock logic, of parts and 
connections, consider the routine of cutting a piece of wood. Taking a step back, imagine 
your actions associated with a woodworking project - all in the confines of your garage. 
Your project is nearing completion as you search for a short piece of wood. The closest 
piece you can find is a tad too long, necessitating the need to make it shorter. In rapid order, 
you measure the length that is required, mark the wood to cut it, and get ready to start the 
electric saw. Before the piece is shortened, I look at the top face of the piece to see “how 
many lines are drawn completely cross the face, top to bottom?” 

“How many lines would you draw?” 


Question : Cutting Wood 

Given a piece of wood that will be cut 
into 2 pieces.... 



how many lines will be drawn across 
the top face before the cut is made ? 
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QUESTION: CUTTING WOOD 


Hopefully, to no surprise, the most frequent answer (with bare exception) is “one 
line”. The term I use for the thinking that results in this answer is “1-line” thinking. 






QUESTION: CUTTING WOOD 


Is “1 line” the correct answer? If not, “What other answer could there be?” While it 
is not immediately obvious, other answers might be “2 lines” or “3 lines”, if not more. The 
terms I use for the thinking that results in these answers are “2-line” thinking, “3-line” 
thinking, and so on. 

What is a possible explanation for the “1 line” answer? Surely, it is but one of many 
possible answers. Could it be that we would draw one line out of habit? Why is the habit not 
“2 lines”? I would offer that the “1 line” answer is an indication of a strong intuitive sense of 
water logic - knowing what the piece of wood is “part of’, knowing where “it came from”, 
and knowing where it “will lead to”. Such a perspective is likely when one is involved in a 
home project and connections are visible as well as conscious. How visible are the 
connections in a work setting, where the connections are less visible? 
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RELATIONSHIPS 


I would like to now offer an explanation for the habit of “1 line” thinking when 
cutting a piece of wood for a home project (step 1). Consider the x-y axes shown, with a part 
characteristic as the horizontal axis and the “cumulative negative impact to others 
downstream” (steps 2 and 3) as the vertical axis. This “negative impact” to others is 
reflective of Dr. Taguchi’s concept of defining quality as the “minimum value of loss 
imparted to the society”. Could it be that we intuitively appreciate the implications of “loss 
imparted” and act to minimize this loss (this impact) when we are the next person in the flow, 
as if receiving our own work? Might we act with a sense of water logic? I believe so. Could 
it be that we are not as particular and will not focus on target, but rather on “meeting the 
requirements of a tolerance” when we are not the next person in the flow? Without the sense 
of a connection, might we act with “rock logic” and default to a part perspective? I believe 
so. 




Cumulative 
Negative 
Impact to 
Others 
Downstream 


TARGET 
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INVESTMENT THINKING 


A stitch in time that saves nine. An ounce of prevention that is worth a pound of 
cure. What these two adages have in common is an awareness of connections - a sense of 
water logic. Notice also that the pro-action, the stitch and the ounce, are far cheaper than the 
nine stitches or the pound of cure. To act in this manner, with a consciousness of 
connections, is to practice the economics of “investment thinking”. 

The general attributes of investment thinking are an allocation of resources (time, 
money, etc.) to prevent a greater expenditure of resources, or to cause a greater gain in 
resources. Both scenarios are heavily dependent on water logic. Lacking consciousness of 
connections, as in a rock logic view of activities, such investment opportunities would be 
overlooked. Yet another reminder of the need for better thinking. 



■ Seeing connections 

m Spending $ to save $ 

• Spending time to save time 

■ Spending resources to save resources 
e Examples 

■ college education, roof repair, time with 
kids 
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PICKING' UP NAILS 


Spending time to save time, as in picking up nails to prevent the flat tire. Another 
example of investment thinking. To do so is to “minimize loss to society” and be reminded 
of Dr. Taguchi’s concept of quality. Another reminder that the discovery of “investment 
opportunities” is limited by our ability to see connections - to appreciate water logic. 



Spending lime (yours) 

to 

Save time (others) 

Minimizing Loss to 
Society 
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SEEING ORGANIZATIONS AS SYSTEMS 


In 1950, Dr. Deming was invited to Japan to meet with engineers and executives from 
hundreds of companies. As an introduction to an 8-day course in quality control he 
introduced his audience to the concept of “seeing organizations as systems”. He encouraged 
them to see their organizations with a sense of flow, of water logic, and be mindful of the 
continuity in the connections between the activities. 

Organizations that practice investment thinking on a continuous basis will generate a 
continuous series of returns of resources in exchange for a smaller allocation of resources. 


Seeing Organizations as Systems 


Design 

and 

.redesign 


Consumer 

research 


Suppliers of 
materials and 

equipment^ and f 

A ^ test of Distribution 

_ materialsprodyctipn, assembly, inspection / . 

c t ? 


J f 

T / 


Consumers 

4 


f // 

Tests of processes 
machines, methods, 
costs 


Source: The New Economics, Dr. W. Edwards Deming 
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UTILIZATION OF THINKING 


A closing reminder of the need to routinely ask questions about connections, to 
practice the principal of water logic. To do so is to prompt investment-thinking decisions. 


§ ■ Where are we going ? 

■ Where does this fit in ? 

■ Where did this come from ? 

■ What is my role ? 

■ What is this part of ? 

■ Where should we invest ? 
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“BHOOMERANG KHARMA” 


A reminder of what might happen when connections are not anticipated, as when one 
acts to focus on the parts and underestimate inevitable connections. 



Figure 20 
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THE ROLE OF BETTER THINKING 


In keeping with the theme of water logic, I offer a theory on the role of better 
thinking. I believe that if we act to “increase awareness on thinking” (through education 
programs or mentoring activities), and then we will “change the way we behave” (when 
connections are better understood, the actions that trigger undesirable results will be altered). 
Subsequently, we will “change the way we work together” (as when we pass on to others 
only what we would pass on to ourselves). In turn, we will then “change the way we run the 
organizations” (to treat others as we would treat ourselves is to change the operation of the 
organization.” Such behaviors will have a reinforcing effect on “increasing awareness on 
thinking”, leading to higher and higher levels of system consciousness and “working 
together”. 



Increase awareness on thinking 


i * • 



Change the way we behave 


Change the way we 
work together 

Change the way we run 
our organizations 
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3-D DIFFUSION EQUATIONS 


Dr. Deming’s last book, The New Economics , introduced his refinements on his 
evolving theory of management. He termed this management theory “the system of 
profound knowledge”, or SOPK as an acronym. This system includes four parts, all related 
to each other. The parts include an understanding of variation, an appreciation of systems, of 
psychology, and of the theory of knowledge. Deming stressed the need to connect the pieces 
as a system. 

As a complement to the partial differential equation that governs the diffusion of heat 
within an object, consider the corresponding equation that would represent the diffusion 
model for profound knowledge (PK) within an organization. In this case, the “Q” term 
represents an education system that generates PK within the organization. Such an education 
system has been developed with The Boeing Company to foster the “better thinking” that 
will lead to reduced “losses to society” and better “opportunities for companies”. 


3-D Dif fusion Equations 

of Heat 

d 2 T/ d x 2 + d 2 T/ d y 2 + 3 2 T/ d z 2 

+ Q’”/k= 1/a a T/ at 

of Profound Knowledge 

a 2 pk/ a x 2 + a 2 pk/ a y 2 + 3 2 pk/ a z 2 

+ Q’” [k = 1/a a PK/ at 
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Outline 


♦ Historical Model for Technical Evolution 


Current Maturity of Nondeterministic Applications 


Future Implementation Trends 



Historical Model for 
Technical 
Evolution 
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Both Science & Probabilistic Methods Seek to Model 
the World Around Us 



5 
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Figure 5 


Both Science & Probabilistic Methods Seek to Model 
the World Around Us 
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Figure 6 
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Likelihood 

Uncertainty 


Conclusive 

Certainty 


Both Science & Probabilistic Methods Seek to Model 
the World Around Us 
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3nts in Detecting, Measuring & Modeling 
^requisites to Technical Advancement 


IihWS 


♦ Detect Phenomena 

♦ Measure Phenomena 

♦ Model Phenomena 


Figure 7 
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Aristotle’s Theories of Mechanics were Constrained by 
Limits of First Crude Clocks 



Clepsydra 325 B.C. 5 Min/Hr Accuracy 
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Figure 9 


Aristotle’s Theories of Mechanics were Constrained by 
Limits of First Crude Clocks 



•• Early water clock 

Clepsydra 325 B.C. 5 Min/Hr Accuracy 


Velocity^ v 151b 


Free Falling Bodi 


accelerate, but heavier Velocitu _ v 

bodies fall faster,” veioc.ty 5 _ v 


Aristotle 370 B.c. 


4000 

B.C. 


300 

B.C. 


Egyptian 

Shadow 

Clock 


Greek 

Water 

Clock 


T»0 


Figure 10 
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Galileo advanced Theory for both Timekeeping & 
Falling Bodies 


1 I 

Velocity^ Velocity 5 = V, 


1883 model of Galileo’s 
1637 pendulum 
escapement design i 



Verge & Spring Galileo 

Foliot Driven Pendulum 

Clock Theory 
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Continued Timekeeping Advancements Drove 
Refinement of Falling Body Equations 



Christiaan Huygens improved Galileo’s 
Pendulum Escapement to achieve 
accuracy of 1 sec/3 Hrs by 1730 
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Moore’s Law has Applied in Principle to Improvements 
in Timekeeping & Falling Body Theories 
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Acceleration — 
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From the Time of the first Mechanical Clocks (1350 A.D.) until 
early 20th Century, Clock Errors were Halved every 50 Years, 
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Current Maturity of 
Nondeterministic 
Applications 


Figure 15 



Figure 16 
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Again... A Long Pause until the Renaissance 



Figure 18 
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Problem of Invariance Was Found for the Second 
Moment Format Method and Resolved in 1970’s 


♦ Second moment format based reliability design became widely accepted although 
a firm logical rationale was not yet demonstrated 

♦ Ditlevsen and Lind independently discovered the problem of invariance in 1973. 

Cornell’s index was not constant when certain simple problems were 
reformulated in a mechanically equivalent way. 

♦ In 1974, Hasofer and Lind initiated the era of modern probabilistic design theory 

by defining a generalized safety index that is invariant to mechanical formulation. 

♦ Several codes developed & implemented in short succession, and routine 
procedures documented in guideline reports 

- Comite European du Beton, 1976; Construction industry Research and 
Information Association, 1977; Canadian Standards Association, 1981 

♦ Uniform Building Code accepted Probabilistics for Civil Engineering applications 

Pascal's 
Theory of 
Probability 
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Gauss & 
Laplace 
Applications 


Safety First Invariance 
Factor Structural Problem 
Approach Codes Resolved 


Movie Clip (MPEG) 


USAF & Dr. Jack Lincoln use Probabilistic Risk Assessment of 
Durability & Damage Tolerance to Recertify Fleet Lifetimes 



• Design must maintain the minimum reliability level, R mln 

• Design must exceed the minimum life expectancy, T min 

• Without NDE, total expected life, T lifcI (where T mo1 < T mln ) 

• With NDE, total expected life, T ll(e2 {where T llle2 > T min ) 
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Figure 20 
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GE Six Sigma Cost and Benefits 


Mostly variable cost productivity and asset utilization 


Up front investment and staying power 
Significant impact on the bottom line 


> Erl racist! from Ur. Norman H. Kuehar's “Design for Six Sigma: Non-DetermlnisUo Design at GE" 
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PADS Use In NASA Programs has Improved System Weight & 
Reliability, Analysis Cycle Time, Test Cost and More 


EELV Cryogenic 
Upper Stage Design 


Mir/Shuttle Docking 
Capture Probability 


NRA8-12 
8’ Tank Redesign 
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Reduce Overall 
Weight by 20% 
Pascal’s 
Theory of 
Probability 
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Future 

Implementation 

Trends 


Figure 23 


Where Do We Go From Here? 

The Vision of Probabilistic Approaches 
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• Reliable products are Achievable 

• Revolutionize Business & industry 

* HxIracted'modKfBiJfrom Mr. Norman R. Kuotiar'a "Design for Six Sigma: Non-Determinisfic Design at GE” 
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12 Steps in the ConsensusBuilder® 
Decisionmaking Process 


Prepare the 

I iecision-Mokfrs 4 *—^. 2 J 


Develop 

Criteria 


Generate 

Alternatives 



Evaluate 
Alternatives 
Using Criterl 






1 : Decision-maker risk 
2: Organizational risk 
3-7: Envisioning risk 
6: Ideation risk 
9-11: Evaluation risk 
1,2,12: Strategic risk 
12: Realization risk 


Srf 


Copyright David G. Ullman, Design & Decision Support 










Success of Probabilistic Approach Requires 
Practitioners to Push the Application Envelope 


Infinite number 
of deterministic 
factors 


Not Yet 
Possible 


Future 

Many random factor: 


Computational 

Difficulty 


Quality or Reality of Design 
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Ultimately, Probabilistic Methods are Aids to 
Decisionmaking by Management 

♦ Goes directly to the Challenge of Modeling the World that We see 
around Us 

♦ Facilitates the Historical Transition from Inferential to Deductive 
Assessments 

♦ Models transform raw Data into Knowledge to enable Decisions 

♦ Technical Methods are being infused Today into Management and 
other inherently Subjective fields 

♦ Managers are crying out for reliable Decision Aids 

♦ Rate of Transition & Successful Use of Probabilistic Methods will be 
determined by Developers of the methods 

CARPE DEUM - Seize the Moment 


Figure 3 1 




Summary of NDA Activities of Professional Societies 


Dr. Suren N. Singhal 
QSS Group Inc. 

NASA Glenn Research Center 
Cleveland, OH 44149 



SUMMARY OF NDA ACTIVITIES OF PROFESSIONAL SOCIETIES 


This presentation summarizes activities of some professional societies in the area of 
non-deterministic approaches (NDA). Major NDA activities of SAE (Society of Mobility 
Engineering) and AIAA (American Institute of Aeronautics & Astronautics) are discussed. 
There are on-going NDA activities in other societies such as ASME (American Society of 
Mechanical Engineering) and ASCE (American Society of Civil Engineering), which are not 
covered in this presentation. The presentation demonstrates that engineers can achieve full 
potential as contributors in professional engineering organizations. 

The SAE G-ll Probabilistic Methods Committee provides a good source of 
information wrt various technological, applications, communications, education, training, 
and related issues for any one to make use of them. 


Summary of NDA Activities 
of Professional Societies 


by Suren Singhal on behalf of all G-l 1 Members 

Presented at The Training Workshop on 
Nondeterministic Approaches and Their Potential 
for Future Aerospace Systems 
NASA Langley, May 31, 2001 



The Engineering Society for Advancing Mobility 

LAND - SEA - AIR - SPACE 


Reach for the Full Potential 


Figure 1 
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OUTLINE 


Non-deterministic approaches are introduced in the context that led to the initiation of 
the SAE & AIAA activities about a decade ago. The SAE, AIAA, and other non-profit NDA 
activities are then discussed. Conclusions & recommendations for the engineering 
community are presented. 


Outline 

• Introduction 

• SAE Activities 

• AIAA Activities 

• Other Professional Activities 

• Conclusions & Recommendations 


Figure 2 




INTRODUCTION 


The critical issue in todays fast moving engineering world is delineated along with a 
proven solution. The accompanying challenges are highlighted. The solution requires a 
systems perspective to fully address the issues of today’s engineering community. All this is 
linked with the role of professional societies in addressing such issues, solutions, and 
challenges for the benefit of all. 


Introduction 

♦Issue, Proven Solution, Challenges 
♦Examples 

♦Systems Perspectives 
♦Role of Professional Societies 


Professional Societies Serving the Community Needs 


Figure 3 






ISSUE AND PROVEN SOLUTION 


Many countries are at the leading edge of technology while the U.S. national budget 
for cultivating the frontier technologies keeps getting tighter, demanding more-than-ever 
efficient return on investment. New innovative ways must be adopted if they lead to real and 
measurable cost reduction and efficiency improvements. Perceptions alone will not suffice. 
Some form of ND A/probabilistic engineering has been proven to result in significant real and 
measured savings. The time is ripe for prudent implementation of such technologies for the 
benefit of all. 


Issue 

Global competition and the state of U.S. national budget 
mandate the need for new innovative ways of increasing 
efficiency with real and measurable cost reduction. 

Proven Solution 

Some form of probabilistic engineering is currently being 
used by some U.S. corporations, resulting in billions of 
dollars of real and measured savings. 

A sample use of probabilistic engineering by U.S. Air Force 
has demonstrated savings of millions of dollars. 

THE TIME IS RIGHT FOR 
PROBABILISTIC ENGINEERING 


Figure 4 


326 






CHALLENGES 


Like any new innovation, there are multiple challenges that must be addressed heads- 
on, rather than ignoring them, so as to enable successful applications of probabilistic 
engineering. With the phenomenal success of engineering innovations designed by 
deterministic approach, it is not an easy task to convince experts to adapt to non- 
deterministic approaches. And then there must be documented proofs that such approaches 
do offer competitive advantage. Further, there is a need for adequate training, the requisite 
tools, and certification guidelines. And then there are cultural and legal barriers, which are 
being addressed by the SAE activities. The bottom line is such a paradigm shift is easier said 
than adopted. 


Challenges 

• Today’s Safety Factor Approach 

• Show me the proof 

• Training, tools, certification 

• Barriers, legal issues 


Paradigm shift is easier said than adopted 


Figure 5 







EXAMPLES OF PROBABILISTIC ENGINEERING 
WITH DEMONSTRATED COST SAVINGS 


The examples of probabilistic engineering with demonstrated savings include those 
by major organizations such as by Northrop Grumman for fighter wing, Lockheed Martin for 
bird strike, P&W for aircraft cooling, Boeing (previously Rockwell) for Space Shuttle 
Docking Module, and applications of Six Sigma at Motorola, Allied Signal, and GE. The 
probabilistic engineering leads to realistic savings, generally with an order of magnitude cost 
to benefit payoff. 


EXAMPLES OF PROBABILISTIC ENGINEERING 
WITH DEMONSTRATED COST SAVINGS 

• Fighter wing — REDUCED WEIGHT BY 15% (Northrop- 
Gramman) 

• Bird strike on aircraft engine — SAVED LIVES (Lockheed-Martin) 

• Aircraft cooling duct fabrication — SAVED $500K (P&W) 

• Space Shuttle docking module — REDUCED TESTING COST 
FROM $500K TO $50K (Boeing-Rockwell) 

• PE-based Design for Six Sigma — MOTOROLA SAVED $1 IB 
and GE ON THE WAY TO SAVE $8B 

Probabilistic engineering is for real with proven 
order of magnitude savings. Expect > 1 to 10 cost 
to benefit payoff!! 


backup 
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SYSTEMS PERSPECTIVE 


The uncertainties are naturally present in every aspect of engineering life cycle from 
defining mission-reliable requirements which must lead to an innovative concept to risk- 
averse design resulting in a competitive product that a customer will value for performance 
over cost and use the product safely with economical maintenance and final retirement with 
overall greater than one return on investment. This is the overall systems perspective that 
weaves uncertainties in each step of any system’s life cycle that would result in profitable 
ventures 


Systems Perspective 


Requirements - — y Concepts 


Mission-Reliable 


Innovative 


ROI 

Retirement 


Multi-Disciplinary 
Analysis, Design & 
Manufacturing 

Risk Averse 


Product 

Competitive 


Maintenance (, — Operation ^ — Customer 

Economical Safe Cost vs. Performance 

Uncertainties are inherent in every step 
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ROLE OF PROFESSIONAL SOCIETIES 


The professional societies can and must act as a catalyst in bring together new ideas, 
technologies, and people. The professional societies can play a pivotal role in bringing to 
limelight worthwhile futuristic innovations faster than they otherwise would. These societies 
must help engineering community make aware of such new technologies, help enhance their 
understanding of convincing intricacies, and be a clearinghouse of much-needed training, 
tools, and experts. The professional societies can accelerate implementation of futuristic 
technologies, starting from small pilot projects conducted in a cooperative multi- 
organizational environment. 


Role of Professional Societies 

• Awareness 

• Understanding 

• Resources 

*Tools 

^Training 

^Experts 

• Implementation 

Professional societies can be the catalyst in bringing 
people & new ideas together 
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SAE G-ll ACTIVITIES 


The SAE G-ll Division activities will be discussed in three parts: (1) the Reliability, 
Maintainability, Supportability, and Logistics (RMSL) Division, (2) the Probabilistic 
Methods (PM) Committee, and (3) the Probabilistic Methods Leadership Council (PMLC). 
The website for details on these activities is provided below. 


SAE G-ll Activities 

♦RMSL Division 

♦Probabilistic Methods (PM) Committee 
♦PM Leadership Council 

SAE G-ll Web site: 

http ://forums . sae.org/access/dispatch.cgi/TEAG 1 1 PM_pf 


Figure 9 



WHY ARE WE HERE? 


The SAE G-ll RMSL Division gathers together semiannually to serve the needs of 
the land, sea, air, and space engineering community. By understanding the needs and 
working on need-based projects, the SAE members deliver information, standards, education, 
and training. 


RMSL 

Division 


Whv Are We Here? 


Information, Standards, 
Education, Training 


atari* 




70,000 SAE Members 


Land, Sea, Air, Space 
Community 
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WHY ARE WE HERE? 


This chart summarizes the benefits of attending the G-ll meetings including: (1) 
face-to-face interaction with peers, (2) an opportunity to measure how an individual 
organization stack up against the state-of-the-art, (3) the best engineering practices, (4) 
technology exchange, (5) networking, (6) list of needed information and resources such as 
meetings, seminars, etc., and (7) a venue for partnering with others. The G-ll Division 
hopes to always provide benefits outweighing the expense of attending the meeting. 


r msl Why Are We Here? 

Division 

•Industry, govt., academia face-to-face 
•How does your organization compare? 
•What are the best practices? 


•Technology interchange and networking 
•Access to information and resources 


•Partnership with some of the best in the business 



Figure 1 1 




HOW DO WE WORK? 


This chart depicts the real workings of the G-ll Division. We try to understand the 
generic needs of organizations at large. We develop projects based on these needs and 
deliver the results to the customer. Of course, the individuals and organizations attending the 
meeting tend to benefit more as their needs are brought out in the open discussions at our 
semiannual meetings. 


RMSL 

Division 


How do We Work? 






Understand needs of Industry 
and individual organizations 

•Division meets twice a year 
•Committee/Project Leaders conduct telecons 

Delivering to the Engineering Community 


Figure 12 






ORGANIZATION 


This chart shows the width and depth in terms of the projects and people involved in 
our organization, which remains dynamic and flexible, based on changing needs of the 
engineering community that it serves. 


























WHAT HAVE WE DONE SO FAR? 


The G-ll Division has published documents with recommendations, guidelines, and 
standards. It has conducted workshops as needed by the community. The Division meetings 
have led to significant industry, government, and academia interaction. The Division is 
dedicated to making a difference. 


rmsl What Have We Done So Far? 

Division 

•Published resource documents, information reports, 
standards and guidelines on RMSL & PM 

•Conducted Workshops 

•Facilitated significant industry, government and academia 
interaction 








PRELIMINARY LIST OF PUBLICATIONS ISSUED 
(AVAILABLE FROM SAE - 724-776-4841) 


A sample list of G-l 1 publications is attached here. 


RMSL Division 


Preliminary List of Publications Issued (Available from SAE - 724-776-4841) 



Title 

Publ. 

Product Code 

Sponsor 

1 

“Evaluation Criteria for Reliability Centered Maintenance (RCM) 
Processes” 

8/99 

JA1011_199908 

TEAG11SL (Chair: D. Netherton) 

a 

“Software Support Concept” 

6/99 

JA1006J 99906 

TEAG11 (Chair: D. Peercy) 

3 

"Reliability Program Standard Implementation Guide” 

3/99 

JA1000/1-199903 

TEAG11R (Chair: D. Elters) 

4 

“Perceptions and Limitations Inhibiting the Application of 
Probabilistic Methods” 

12/98 

AIR5086 

TEAG1 1PM (Chair: C. Pomffet) 

m 

“Software Reliability Program Standard” 

7/98 

JA1002_1 99807 

TEAG11SW (Chair: D. Peercy) 

a 

“Software Supportabiiity Program Implementation Guide” 

7/98 

JA1004J99807 

TEAG 1 1SW (Chair: D. Peercy) 

a 

“Reliability Program Standard” 

6/98 

JA 1000- 199806 

TEAG11R (Chair: D. Elters) 

L 

Probabilistic methods, A Joint Industry/Government/ Academia 
Assessment of Needs and Goals” 

10/97 

ARD050047 

TEAG 11PM (Chair: S. Singhal) 

m 

“Software Supportabiiity - An Overview” 

1/97 

AIR5121 

TEAG11SW 

a 

“Integration of Probabilistic Methods into the Design Process” 

1/97 

AIR5080 

TEAG11PM (Chair: E. Fox) 

m 

“Reliability and Safety Process Integration” 

7/96 

A1R5022 

TEAG11 

ri 

“Solid Rocket Booster Reliability Guidebook- Vol. 11 Probabilistic 
Design & Analysis Methods for Solid Rocket Boosters” 

6/96 

AIR5006/2 

TEAG 11 

m 

“Liquid Rocket Engine Reliability Certification” 

4/96 

ARP4900 

TEAG11 

a 

Recommended RMS Terms and Parameters” 

12/95 

AIR4896 

TEAG11R 

a 

“RMS Information Sourcebook” 

11/93 

ARD50046 

TEAG11 

r 

“The FMECA Process in the Concurrent Engineering (CE) 
Environment” 

6/93 

AIR4845 

TEAG 11 

a 

“Solid Rocket Booster Reliability Guidebook” 

2/91 

ARD50013 

TEAG11 

□ 

“Survey Results: Computerization of Reliability, Maintainability 
& Supportabiiity (RM&S) in Design” 

1/90 

AIR4276 

TEAG 11 

backup 


Figure 15 


337 

















































































PRELIMINARY LIST OF PUBLICATIONS IN PROGRESS 
(DRAFTS MAY BE AVAILABLE FROM CHAIRPERSON) 


A sample of list of publications in progress is attached here. 


RMSL Division 

Preliminary List of Publications in Progress (Drafts May Be Available From Chairperson) 



Title 

Publ. 

Product Code 

Sponsor 

1 

“Maintainability Program Standard 


JA1010 

TEAG11M (Chair: W. Gregory) 

2 

“Basic Concepts, Models and Approximate Methods for 
Probabilistic Engineering Analysis” 

■ 

AIR5083 

TEAG1 1PM (Chair: D. Ghiocel) 

H 

“Applications of Probabilistic Methods” 


AIR109 

TEAG11PM (Chair: T. Tomg) 

r 

“Legal Issues Associated with the Use of Probabilistic Design 
Methods” 

■ 

AIR5113 

TEAG1 1PM (Chair: A. Pickard) 

m 

“Reliability Testing Standard” 


JA1009 

TEAG1 1R (Chair: W. Grimes) 

a 

“Failure Modes, Effects, and Criticality Analysis Procedures” 


12336 

TEAG11S (Chair: H. Hetrick) 

m 

“Supporrtability Process Standard” 


J2336 

TEAG1 IS (Chair: H. Hetrick) 

a 

“Guide to the Reliability Centered Maintenance (RCM) Standard 


JA1012 

TEAG1 1SL (Chair: D. Netherton) 

m 

“Software Supportability Program Implementation Guide” 


JA1005 

TEAG11SW (Chair: D.Peercy) 

m 

“Software Reliability - An Overview” 


J2443 

TEAGUSW (Chair: D.Peercy) 

m 

“Software Reliability Program Standard” 


J2444 

TEAG11SW ( Chair: D. Peercy) 

m 

“Software Reliability Implementation Guide” 


J2445 

TEAGUSW ( Chair: D.Peercy) 

m 

“Software Supportability Program Standard” 


J2446 

TEAGllSW (Chair: D.Peercy) 


“Software Supportability Implementation Guide” 


J2447 

TEAGUSW (Chair: D.Peercy) 

19 

“Software Support Concept” 


J2448 

TEAG1 1SW ( Chair: D. Peercy) 
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WHERE ARE WE HEADED? 


The Division will continue focusing on relevant RMSL 
advantage of web-based communication. 


and PM needs, taking 


RMSL Division 


Where Are We Headed? 


•RMSL should remain the focus unless otherwise 
so indicated by our customers. 


•Need to revitalize and reinvigorate all G-ll activities 
and participants based on customer needs. 

•Transition to an electronically-linked network to rapidly 
respond to individual and organizational needs, but 
continue face-to-face semi-annual meetings. 

•Elevate G- 1 1 to Systems Engineering Council 


Just do what’s relevant & will be useful 


backup 
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REVITALIZATION OF G-ll 


This chart outlines the Division vision and goals with focus on meeting customer 
needs. Our members invest hundreds and thousands of volunteer hours. We want to make 
sure we work only on what is relevant, needed, and will be useful to our customers such as 
you, the reader. 


RMSL Division 

Revitalization of G-ll 

Vision: 

Be the authoritative source of RMSL information, 
education, and standards that the national and 
international leaders turn to! 

Goals: 

(1) 

Re-establish projects based on customer 
need only. (Initial buy-in, continuous 
interest, of direct use and benefit.) 


(2) 

Link projects to participants with overlap in 
their day job. 


(3) 

Communicate with senior management on 
what we do in conjunction with what will 
attract their attention. 



backup 


Figure 18 



REVITALIZATION OF G-ll (CONTINUED) 


Our goals include establishing appropriate liaisons with national and international 
engineering community. 


RMSL Division 

Revitalization of G-ll (Continued) 

Goals: (4) 

Establish liaisons with relevant groups. 
(NATO, U.K., Ministry of Defense, ISO, 
IEEE, NAE, ) 

(5) 

Broadcast relevant standards already 
developed by G- 1 1 . 

(6) 

Meet at locations most likely to attract 
participants. 

(7) 

Need to listen to and hold hands of new 
participants. 

(8) 

Integrate RMSL workshops with RAMS 

backup 
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G-ll PROBABILISTIC METHODS COMMITTEE (PMC) - VISION 


Now let’s talk about the G-ll Division’s PM Committee. The vision was set ten 
years ago and we continue brainstorming, implementing, and delivering the requisite results 
for the benefit of the engineering community at large, with special attention to the 
organizations to the attending members. 


G-ll Probabilistic Methods Committee (PMC) 

Vision 

To serve as the premier Probabilistic Methods group 
with balanced, broad representation in industry, 
government, and academia that carries with it 
authoritative insight and the ability to envision, 
initiate, and implement a holistic agenda for 
probabilistic methods that benefits all people. 


Brainstorm, initiate & implement probabilistic projects 
for the benefit of all, especially member organizations 






G-ll PROBABILISTIC METHODS COMMITTEE (PMC) - PEOPLE 


This chart shows the people of the PMC organization involved in technology 
development, technology applications, communications, issues such as legal ones, and new 
initiatives. 


G-ll PMC People 
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G-ll PROBABILISTIC METHODS COMMITTEE (PMC) - PRODUCTS 


The PMC products include documents, education, training, workshops, 
recommendations, guidelines, and standards. 


G-iiPMc Products 

•Technology Development & Applications - Compile Information 
•Documents (AIR/ARD) 

•Education & Training 

•Recommendations to industry, government, and academia 
•Standards 

G-ll produces information, documents, education, 
training, recommendations, and standards 


Figure 22 





G -11 DIVISION PROJECT INFORMATION - MIAMI, FLORIDA 



G-ll PMC 


G -11 DIVISION 
PROTECT INFORMATION 
UPDATED AT MARCH 26-28. 2001 MEETING 
MIAMI. FLORIDA 


NAME OF PROJECT 

LIST OF PARTICIPANTS: 

(please include e-mail address) 

This list will be published on the web page for this 
project. It will also serve as a special access list for 
the Team’s Private Area located in SAE’s Private 
Forum. This will be where draft documents reside 
for this project and allow easier communication 
among team participants. 

NOTE: INDICATE PRIMARY (P) OR 
SECONDARY (S) 

AIR/ARD NUMBER AND TITLE: 
SCOPE/PURPOSE/END RESULT: 


End Result: 


Please return this form to Suren Singhal or Eric Fox before leaving Meeting 
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G -11 DIVISION PROJECT INFORMATION - RENO, NEVADA 


We further show the amount of critical information that we collect for each project. 
Relevance to industry is a MUST. 


G-ll PMC 


G -11 DIVISION 
PROTECT INFORMATION 
UPDATED AT OCTOBER 23-26, 2000 MEETING 
RENO. NEVADA 


TABLE OF CONTENTS: 

(If a draft is available, it will be placed on the 
web page for the project.) 


RELEVANCE TO INDUSTRY/GOV’T: 
(who is going to benefit) 


PROJECTED COMPLETION DATE: 



Please return this form to Suren Singhal or Eric Fox before leaving Meeting backup 
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G -11 DIVISION PROJECT INFORMATION - RENO, NEVADA 


We conclude each project meeting with future plans for the next 6 months. Our 
members continue working on these projects along with their day jobs. Monthly telecons 
with Project, Subcommittee, Committee, and Division chairs ensure flow of information, 
continuity of work, and high motivation among active members. 


G-ll PMC 


G -11 DIVISION 
PROTECT INFORMATION 
UPDATED AT OCTOBER 23-26. 2000 MEETING 
RENO. NEVADA 


MEETING ACCOMPLISHMENTS: 


FUTURE PLANS: 

(Action Items/Including Dates) 



Please return this form to Suren Singhal or Eric Fox before leaving Meeting 


backup 
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MISSION STATEMENTS FOR OUR PM COMMITTEE WEBSITE 


The mission of some of our PM technology project is described on this chart. 


G-ll PMC 

MISSION STATEMENTS FOR OUR PM COMMITTEE WEBSITE 

Subcommittee: 

Technology 

Mission: 

To develop and disseminate technical information about probabilistic 
Methods which can be used easily by industry, government, and academia. 

1 . 

Project: 

Integration of probabilistic Methods in Design 


Mission: 

To develop an approach which will integrate probabilistic methodologies 
with design practices, procedures, and software codes currently being used. 

2. 

Project: 

Computational Probabilistic Methods 


Mission: 

To create a state-of the-art, nationally recognized resource document 
on Probabilistic methods for use by industries for advanced 
engineering applications and probabilistic designs. 

3. 

Project: 

Applications of Probabilistic Methods 


Mission: 

To capture previous experience and lessons learned in the application of 
probabilistic methods, and to provide examples and points-of -contact 
for initiating new applications. 

4. 

Project: 

Probabilistic methods Case Studies 


Mission: 

To provide guidelines by which probabilistic methods should be used in 
different types of problems. 

5. 

Project: 

Integration of probabilistic methods in Manufacturing 


Mission: 

To identify and describe the engineering challenges, requirements, and 
methods employed in manufacturing and quality control. backup 
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TECHNICAL SUBCOMMITTEES AND PROJECTS - COMMUNICATIONS 

The mission of some of our communications projects is described on this chart. 


G-ll PMC 


Technical Subcommittees and Projects 
COMMUNICATIONS 


To identify the industry need and means of rapid communication and transfer of the probabilistic technology to the industry and 
facilitate the adaptation of the requisite technology by the industry. 

Projects 

1. Needs/Goals 

To identify industry, government, and academia needs and goals and to ensure SAE G-ll PM Committee addresses these needs 
and goals. To promote PM usage in industry and government through (a) increased awareness by providing pre-eminent 
source of information on all aspects of PM, and (b) induced synergism by establishing communications between 
organizations/parties interested in PM. 

2. Workshop 

To develop and present a workshop demonstr ating practical applications of PM. 

3. Newsletter 

To communicate G-ll and other nationai/intemational PM activities via a semi-annual newsletter. 

4. Membership 

To expand participation of scientists, engineers, and managers in G-l 1 PM activities. 

5. Publications 

To make people aware of PM technology and its potential benefits by publishing articles in engineering and non-engineering 
magazines. 

6. Awards 

To recognize significant industry, government, academia PM contributions exemplifying time and cost savings, support, 
training, and dedication. 

7. Website 

To create and update a website location to inform the public of G-l 1 PM technology and its potential benefits via an electronic 
environment. 


8. G-ll Liaison 
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TECHNICAL SUBCOMMITTEES AND PROJECTS - COMMUNICATIONS 


The mission of some of our issues projects is described on this chart. 


G-11PMC 


Subcommittee: 

Issues 

Mission: 

To address the controversies, reluctances, litigation aspects and 
standards associated with the introduction of PM into design, 
manufacturing, certification, operation, maintenance, and retirement. 

1. Project: 

Barriers to probabilistic Methods 

Mission: 

To address the barriers which impede the acceptance of PM in the 
design, manufacturing, and user communities and examine the benefits 
and limitations of PM so that their use can be properly understood and 
practiced. 

2. Project: 

Probabilistic Methods Legal Issues 

Mission: 

To address the barriers which impede the acceptance of PM in the 
design, manufacturing, and user communities and examine the benefits 
and limitations of PM so that their use can be properly understood and 
practiced. 

3. Project: 

Probabilistic Methods Legal Issues 

Mission: 

To examine the legal aspects of utilizing PM, most notably the 
quantification of risk/safety and the attendant ramifications. 

Subcommittee: 

New Initiatives 

Mission: 

To initiate new projects with significant potential impact on use and 
communication of PM technology. 

backup 
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ACCOMPLISHMENTS 


This chart is not to beat the drums but as a tribute to the hard and dedicated work of 
our members that have led to accomplishments that they can be proud of. 


g- 11 pmc Accomplishments 

•In 1992, we began with 6 members with a goal of 50 in 5 years. 

Nine years later today, we stand at > 100 (including non- 
attending ones)! 

•In 1993, we began with 5 generalized long term goals. 

Eight years later today, we stand at 20 (15 active) projects! 

•In 1994, we began working on 1 SAE document. 

Seven years later today, we have published 3, are about to 
publish 3 more, and are pursuing 4 more. 

•In 1995, we began with the idea of PM Leadership Council. 

Six years later today, we have > 30 Council members! 

•In 1996, we began with an idea of a PM newsletter. 

Five years later today, we have published 9 issues! 

backup 
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ACCOMPLISHMENTS 


Our members must continue working with the same vigor for at least a decade more, 
as they have in the last decade. 


G ' 1 1 PMC Accomplishments 

•In 1997, we introduced 4 PM achievement awards. 

Four years later today, we are preparing for the 5 th award 
ceremony! 

•In 1997, PMLC recommended we conduct PM Workshops. 

We presented PM Workshops in 1997 & 1998! 

•In 1999 and 2000, we focused on & demonstrated stable growth 
in the PM attendees & enhanced our linkage with industries. 

•In 2001, we are beginning with more bold ideas!! 

We are influencing our organizations’ competitiveness! 
With your dedication, anything is possible!! 

backup 

Figure 30 
352 





STATUS OF DOCUMENTS 


A status of some of our current documents is delineated here. 


G-ii pmc Status of Documents 


Category 

Title 

%Complete 

Estimated 
Completion Date 

SAE Report # 


Probabilistic Engineering 

99% 

10/1 

AIR 5083 


Methods, Volume 1 





Probabilistic Engineering 

75% 

1st draft by 10/01 

Not Yet Assigned 

Technology 

Methods, Volume II 




Numerical Review 

75% 

1st draft by 10/01 

AIR 5110 


Input Distribution Selection 

5% 

Outline by 10/01 

Not Yet Assigned j 


Probabilistic Reliability 

20% 

1st draft by 10/01 

Not Yet Assigned : 


Application Cases 

80% 

(Volume 1) 

Final by 10/01 

AIR 5109 

Applications 

Airworthieness 

70% 

1st draft by 10/01 

Not Yet Assigned 


Manufacturing 

40% 

In Progress 

Not Yet Assigned 




3/03 



Legal Issues 

99% 

Approved 

AIR 5113 

| 

Guidelines 


Discussion Phase 
10/02 

AIR 5115 

Issues 

Minimum Competency 

40% 

10/1 

Not Yet Assigned 


Diagnostics 


Just Beginning 
10/03 

Not Yet Assigned 

t - 

Flight Test Cost Reduction 

5% 

Outline by 10/01 

Not Yet Assigned 
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FUTURE PLANS 


The future of the SAE G-ll PMC is to continue the path it embarked upon about ten 
years ago, i.e., keep working until PM becomes a routine practice. 


Gui pmc Future Plans 

G-ll PMC as an internationally recognized premier source for: 

•PM Information 

•PM Experts 

•PM Applications 

•PM Training 


Keep working until PM becomes a routine practice! 


Figure 32 



PROBABILISTIC METHODS LEADERSHIP COUNCIL 


This chart describes the charter and focus of the PMLC, an organization that took 
birth about 5 years ago to serve as the senior advisory council to the SAE G-ll PMC and to 
encourage the implementation of PM in the industry. The Council is comprised of senior 
executives from industry, government, and academia. 


Probabilistic Methods Leadership Council 

• Charter - High-Level Advisory Group 

• Members - Senior Executives 

• Current Focus - Risk Assessment & Probabilistic Design 
Practice 

• On-Going Projects - Recommend minimum PM competency 
to engineering accreditation board 


Leadership Council has made a difference in 
accomplishing the G-ll PMC vision. 


Figure 33 
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AIAA ACTIVITIES 


This chart lists the two AIAA activities. The PM subcommittee of the Structures TC 
that started about ten years ago is now focused on service life design and reliability 
assessment. About three years ago, the Non-deterministic Approach (NDA) Forum was 
established to serve the growing need of the engineering community. 


AIAA Activities 


•Technical Subcommittee on Service Life 
Design & Reliability Assessment & the 
NDA Forum 

•Working Technical Group - 
Nondeterministic Approaches (NDA) 


AIAA Structures TC web site: 

http://jafar.ncsa.uiuc.edu/aiaa/organization/TechSub/reliability.html 
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TECHNICAL SUBCOMMITTEE ON SERVICE LIFE DESIGN 
& RELIABILITY ASSESSMENT 


This chart outlines the AIAA activities in the area of PM and NDA, mostly focused 
on panel discussions and paper presentations, unlike the SAE G-ll activities, which are three 
days of, focused interaction on PM with your peers. 


AIAA 

Technical Subcommittee on Service Life Design 

& Reliability Assessment 

•Initiated as Probabilistic Methods (PM) 
Subcommittee of the Structures TC in 1993 

•Initiated & successfully implemented focused 
sessions on PM papers at the annual SDM 
Conference 

•Initiated & have organized a panel discussion at 
the annual SDM Conference. 

•Approved by AIAA as NDA Forum 


The aerospace professional engineering community 
has pulled together to make AIAA activities a success 
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WORKING TECHNICAL GROUP - NONDETERMINISTIC APPROACHES (NDA) 


This chart describes the NDA working technical group. 


AIAA 

Working Technical Group - 

Nondeterministic Approaches (NDA) 

•An electronic committee dedicated to furthering 
the implementation of nondererministic 
approaches in the engineering community 

•Conducted a joint industry/government/academia 
workshop for nation-wide recommendations on 
the use of nondeterministic approaches. 


A dedicated group of members continue to encourage 
the use of nondeterministic approach 





OTHER NON-PROFIT PROFESSIONAL ACTIVITIES 


This chart gives the address of a nonprofit website with a focus on nontraditional 
approaches. The web site will be open soon. NDA and PM will be a major part of this web 
site. 


Other Non-Profit Professional Activities 

A web-based professional community & 
resource for non- traditional approaches: 

WWW.NTACENTER.COM 

•Web site under construction 

•First segment with focus on PM & NDA 
accessible in August, 2001 


A central one-step web-based resource for non- 
traditional approaches for America tomorrow! 
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CONCLUSIONS 


In conclusion, there is evidence out there that the probabilistic approaches will 
provide an order-of-magnitude savings, which can be measured. The SAE G-ll PMC 
provides a forum to bring the people and technology together. There are publications dealing 
with PM technology, applications, and communications issues that may be useful for any 
organization engaged in understanding and implementing probabilistic approach. 


Conclusions 

•Payoff from interdisciplinary probabilistic engineering will be 
orders of magnitude of investment. 

•SAE G-ll PMC provides a forum: 

- to learn from each other 

- to compile & disseminate relevant information 


SAE is fulfilling the current PM need 


Figure 38 
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RECOMMENDATIONS 


On behalf of the SAE G-ll PM Committee, recommendations to adopt the PM 
technology are to first sensitize our engineers to start thinking probabilistically, followed by 
education and access to relevant tools. We must first realize the PM benefits on a small 
project, before going full scale and realizing the full PM potential. 


Recommendations 

•Sensitize & Educate yourself 
•Find the right tools 

•Start with applying PM to the right prototype 
•Realize full potential of PM 


PM - A ROUTINE PRACTICE! 


Figure 39 


361 



YOUR ACTION PACK 


For those interested in joining or just learning more about the SAE G-ll PM 
activities, you may propose a project of your interest for our consideration or submit a PM 
application for publication in our resource documents. The bottom line is - MANAGE 
UNCERTAINTIES OR RISK BEING MANAGED BY THE UNCERTAINTIES - THE 
CHOICE IS YOURS!! 


Your Action Pack 

(1) Get involved in G-ll - Announcement for the next 
G-ll PMC meeting 

(2) Propose your project - New Project executive 
Summary Form 

(3) Submit a PM application for publication - PM 

Application Summary Sheet 

(4) Inform your colleagues - Suggestion for potential 
new members 


Manage Uncertainties OR 
Risk Being Managed by Them! 


Figure 40 
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ACTION (1) GET INVOLVED IN G-ll 
ANNOUNCEMENT FOR THE NEXT G-ll PMC MEETING 


This chart includes a brief description of projects to be discussed at the next G-ll 
meeting. 


Action (1) Get Involved in G-ll 
Announcement for the next G-ll PMC Meeting 

The Fall 2001 Meeting of the SAE G-ll Probabilistic Methods Committee will be held in Monterey, California during October 1-3, 2001. 

The three-day meeting will be focused on technical discussions among your peers from industry, government, and academia. 

The topics to be discussed include: 

(1) Probabilistic Engineering Methods - What are the various probabilistic methods, how are they alike and/or different, where are they applicable, and 
how can you use them in real-life? 

Relevance to Industry & Government - Details and references on various probabilistic methods and recommendations on which methods can be used 
for what real-life problem. 

(2) Numerical Review- Several typical engineering problems are being solved using different probabilistic simulation codes. The discussion includes: 
what problems, what results by different methods, and how can industry use which code for what problem. 

Relevance to Industry & Government - Case studies of typical problems encountering uncertainties, results of solutions to these problems run by 
different codes, and recommendations on which code is applicable where. 

(3) Input Distribution Selection - What distribution to select when there is little or no data? 

Relevance to Industry & Government - Too often, we get bogged dowu thinking we need a lot of data before we can quautify uncertainties. Not True. 
There are ways to do credible probabilistic analysis with little data. 

(4) Application Cases - We are compiling the applications of probabilistic analysis demonstrating time & cost savings by various organizations. 

Relevance to Industry & Government - Too often, we say, “Show Me the Proof of the Pudding”. With help from many contributors, we hope to 
produce such a document Problem is - not too many people are coming forward due to proprietary nature. So, we are asking to document only 
minimum information including problem description, what method used, did it result in any savings, and how much? 

(5) Airworthiness - How to use probabilistic methods for airworthiness - a project proposed by a PMLC Member. 

Relevance to Industry & Government - Airworthiness is a key issue for the aerospace community. There are uncertainties associated with it. By 
learning how to assess the effects of these uncertainties, we hope to be able to help industry produce airworthy vehicles which are more efficient and 
cost effective at the same time. 
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ACTION (1) GET INVOLVED IN G-ll 
ANNOUNCEMENT FOR THE NEXT G-ll PMC MEETING 


This chart continues the brief discussion of the projects to be discussed at the next G 
1 1 PM meeting. 


(6) Manufacturing - This project started with plans for integrating probabilistic methods in the manufacturing process but is currently focused on 
dimensional tolerancing during the manufacturing process. 

Relevance to Industry & Government - Tolerancing during the manufacturing process is a key issue that governs warranty, cost, failure rate, etc. 

With this project, we hope to provide guidance on tolerancing. 

(7) Legal Issues - We are looking at legal precedence and what issues may arise when you use probabilistic methods. 

Relevance to Industry & Government - There is the widespread belief that when things are designed using deterministic approach, they are designed 
correctly. And that if you use probabilistic approach, you designed it to fail (one in so many times). Sure, it invites public scrutiny. The fact is, it is the 
probabilistic approach that accounts for real-life uncertainties allowing us to design correctly. 

A paper was published in an AIA A Conference with an eve-opening conclusion - if an organization does not use probabilistic methods, tools for which 
are now available, then that organization could be find negligent for not using such tools. 

(8) Standards - What standards need to be set by whom, when, etc.? 

Relevance to Industry & Government - Much discussion is taking place in consultation with FAA, industry, and others on how to go by start setting 
a pilot standard for certification by probabilistic methods, eventually leading to full standards for analysis, design, manufacturing, testing, certifications, 
maintenance, operations, and retirement. 

(9) Competency - Wliat is the minimum competency in probabilistic methods that our engineers should have before graduating from college? This project 
was proposed by SAE PMLC. 

Relevance to Industry & Government - We have initiated contact with ABET and are brainstorming as to what should our engineering colleges 
teach, both on the undergraduate and the graduate level so that our industry and government don’t have to spend a lot of money training engineers in 
how to quantify uncertainties. 

( 10) Diagnostics - How to incorporate probabilistic methods into diagnostics? 

Relevance to Industry & Government - Knowing how to account for uncertainties in diagnostics, can lead to significant cost savings and can result 
in reducing failures. 

(11) Probabilistic Reliability - How to compute reliability by quantifying uncertainties? 

Relevance to Industry & Government - Correct reliability computations both at the component and system level are needed so one can design an item 
based on its expected usage and life span. 

(12) Flight Test Cost Reduction - How can one reduce the high cost and time of flight testing? We will look at the whole picture including analysis, 
ground testing, and in-flight testing? This project was inspired by the Boeing President for Phantom Works, Mr. Swain. 

Relevance to Industry & Government - cost savings and faster time to market!! 

There are other ongoing operational projects. If you can make a good case, we will consider a new project that can help our industry and government. 
For farther infonnation, contact: 

Meeting Details: Kerry Tielscb (ktielsch@sae.org) 

Technical: Suren Singbal (ssinghal@grc.nasa.gov) 
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ACTION (2) - PROPOSE YOUR PROJECT 
NEW PROJECT EXECUTIVE SUMMARY FORM 


This chart is the new project executive summary form. You are welcome to complete 
one for the project of your interest and submit it to S AE. 


Action (2) - Propose your project 
New Project Executive Summary Form 

Title: 

Submission Date: Revision: 

Project Leader: Alternate: 

(Address) 

(Phone/Fax) 

(E-mail) 

Background: 

Objective(s): 


Scope: 


Benefit to Industry/ 
Government/ Academia: 


Relation to Other AIR’s: 


Target Dates: Outline- 

First Draft - 

Expected Completion Date - 

When completed, please submit to your committee chairperson. 
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ACTION (3) - SUBMIT A PM APPLICATION FOR PUBLICATION 
PROBABILISTIC METHODS APPLICATION SUMMARY SHEET 


You may use this form to submit an application of probabilistic methods for 
publication in an SAE document 


1 . 

Action (3) - Submit a PM application for publication 
Probabilistic Methods Application Summary Sheet 
Application No: (Do not answer this question) 

2. 

Type of Industry: 

3. 

Project Title: 

4. 

Reason for Using Probabilistic Approach: 

5. 

Probabilistic Method Used: 

6. 

Rationale for Selection of the Type of Probabilistic Analysis Used for This Application: 

7. 

Probabilistic Analysis Results Summary and Benefits: 

8. 

Describe Whether or Not the Results Were Verified (Analytically, or by Test): 

9. 

Potential Application of This Analysis to Other Industries: 

10. 

Cost Versus Benefits Analysis: 

11. 

Referenced Technical Report or Paper: 


Please submit to Suren Singhal at: 
ssinghal @ qssgess.com 
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ACTION (4) - INFORM YOUR COLLEAGUES 
SUGGESTIONS FOR POTENTIAL NEW MEMBERS - PLEASE PRINT 


This page is for sharing with those interested in joining the SAE G-ll PM 
Committee. Hopefully, you may benefit from the work of professional engineering societies. 
Together, we can make a difference. 


Action (4) - Inform your colleagues 
Suggestions for Potential New Members - Please Print 

Name: 

Last: First: 



Company: 


Email: 


Phojie Number; 


Fax#: 


Address: 




Name: 

Last First: 



Company: 


Email: 


Phone Number: 


Fax#: 


Address: 




Name: 

Last: First: 



Company: 


Email: 


Phone Number: 


Fax#: 


Address : 





SUBMITTED BY:. 


Phone#: 

EmailT - 
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Probabilistic Approaches for Evaluating Space Shuttle Risks 


William Vesely 

Space Shuttle PRA Coordinator 
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Probabilistic Approaches for 
Evaluating Space Shuttle Risks 

Bill Vesely 

Space Shuttle PRA Coordinator 
May 31, 2001 

Figure 1 

Objectives of the Space Shuttle PRA 

• Evaluate Mission Risks 

• Evaluate Uncertainties and Sensitivities 

• Prioritize Contributors 

• Evaluate Upgrades 

• Track Risks 

• Provide Decision Tools 

Figure 2 
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Space Shuttle PRA Significance 


• Largest Space Shuttle PRA Effort to Date 

• Direct Participation of the NASA Centers 

• Recognized by the Space Shuttle Program 

• Model Integration Across JSC, MSFC, KSC 

• NASA Headquarters Support and Oversight 

• NASA Computer Code and Models 


Figure 3 


Space Shuttle PRA Participants 

• JSC, MSFC, and KSC 

• NASA Headquarters 

• Prime Contractors-USA, Boeing, Rocketdyne, 
Morton-Thiokol, Pratt- Whitney, Lockheed-Martin 

• Support Contractors- SAIC, HEI 

• Supporting Consultants 


Figure 4 
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Space Shuttle Elements 


• Space Shuttle Main Engines (SSMEs) 

• Solid Rocket Boosters (SRBs) 

• External Tank (ET) 

• Redesigned Solid Rocket Motors (RSRMs) 

• Advanced High Pressure Pumps (ATHPPs) 

• Orbiter 

Figure 5 

Types of Losses 

• Loss of Crew and Vehicle 

• Processing Incurred Losses 

• Mission Aborts 

• Mission Partial Failures 

• Economic Losses 

Figure 6 
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Risk Contributors 


• Hardware Failures 

• Fires and Explosions 

• Human Performance 

• Processing Effectiveness 

• Cracks and Leaks 

• Aging Degradations 


Figure 7 


Program Approach 

• Develop Risk Framework 

• Identify Risk Elements to be Included 

• Establish System Engineering Interfaces 

• Develop Risk Models 

• Assemble Data 

• Quantify Risks and Uncertainties 

• Support Applications 

Figure 8 
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Mission Event Tree Model Structure 



Space Shuttle PRA Flow Diagram 


Figure 9 


Function Definitions for a Space Shuttle Mission for the PRA 

ASCENT Mission Event Tree (7 Top Events) : 


Top Event Names: 

Top#l 

Top#7 

MECH-SUPPT-ASC 

Top#2 

FC-ASC 

Top#3 

STRUCT-INTG-A! 

Top#4 

SC CRW-ENV 

Top#5 

PROPULSION 

Top#6 

ETSEP-SHUTD 

JV ORB- 

Modeling Approach: 

INSRT 

ET 

FT 

FT 

FT 

ET 

ET 

ET 

Modeling Status: 

( developed ) 

(to develop) 

(to develop) 

(to develop) 

(developed) 

(developed) 



(developed) 








Top Event Names: 

Modeling Approach : 
Modeling Status: 


ORBIT Mission Event Tree (5 Top Events): 


MfiClj-SVP PT- QRB FC-ORB STRUCT -INTO 

CRW-TSK 

ET FT FT 

^developed) (to develop) (to develop) 


Top#4 Topff5 

9 CRW-ENV 
FT 

(to develop) (to develop) 


Top Event Names: 

Modeling Approach : 
t Modeling Status: 


ENTRY/LANDING Mission Event Tree (5 Top Events): 


Topffl 

Top#2 

Top#3 

Top#4 

TopttS 

MECII-SUPPT -ENT 

FC-ENT 

STRUCT-INTG-ENT 

CRW-ENV 


rNPQ-pECi 

ET 

FT 

FT 

FT 

ET 

(developed) 

(to develop) 

(to develop) 

(to develop) 

(to develop) 
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Probabilistic Approaches 


Reliability Models 
Phenomenological Models 
Dependency Modeling 
Probability Networks 
Bayesian Probability and Statistics 
Sensitivity Studies and Designs 
Decision Theory Applications 
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Figure 17 


Event Sequence Diagram for Phenomenological Modeling 

(1) Estimated values from Marshall/ JSC/ F-1 8 data. Maybe 
different or same for orbiter/SRB APU. 


(2) Somewhat subjective - White Sands to help 



Figure 18 
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Failure Probabilit 







Failure Probability Density 


Failure Probability Density Versus Time 
STS 39R 

7.00E-06 
6.0QE-06 
5.00E-06 
4.00E-06 
, 3.00E-06 
2.00E-06 
1.00E-06 
O.OOE+OO 

10 12 14 16 18 J 

Time (Seconds) 

TIRE FAILURE PROBABILITY DENSITY PROFILE IN AN ORBITER LANDING 

Figure 21 


Modeling Objectives 

• Consistency Across Diverse Elements 

• Modularized Risk Models 

• Prioritization of Hardware, Humans, and 
Processes 

• Thorough Treatment of Uncertainties and 
Sensitivities 

• Defensible Results and Conclusions 

• Effective Presentations 

Figure 22 
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Program Status 


• Modeling Guidelines Issued 

• Risk Framework Model Developed 

• Risk Elements Defined 

• First Stage Models Being Completed 

• Computer Models Being Constructed 

• Initial Results End of FY01 

• Final Results End of FY02 

• Future Extensions Being Planned 


Figure 23 
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